Unix on Lloyd Rochester's Geek Blog

Understanding Process Execution

Tue, 29 Nov 2022 00:00:00 +0000

Understanding Process Execution

In this post we will dive in to command execution from inside a program. The ability to execute commands from inside a program to obtain their output and exit code is very common for automation and testing use cases. Unfortunately, due to how Operating Systems work it is not a simple process to obtain the output and exit code from a running program.

Remote Process Execution

Take for an example a program that from within we execute a command and capture the output in real time, as well as, capture the exit code once the program ends.

UART Theory and Programming on the Raspberry Pi

Tue, 16 Aug 2022 00:00:00 +0000

UART Theory and Programming on the Raspberry Pi

This post outlines how to control the UART in a Raspberry Pi using the C programming language. We will also explore parity checks and handling. Parity checking was the main driver for this post, however, if you’re looking for C code to control a UART that is also asynchronous please keep reading. This post contains a small command line tool that can asynchronously read and write to standard input and output through two connected UARTs. The command line tool has a number of options to set the speed, tty, parity and number of stop bits.

Understanding memory reallocation with realloc

Wed, 03 Aug 2022 00:00:00 +0000

Understanding memory reallocation with realloc

Let’s better understand how memory is allocated, then reallocated on the heap. When we allocate blocks of memory they are stacked on top of the heap with increasing memory addresses. What happens when we reallocate a memory block to make it larger when it’s isn’t the last item on the heap?

Heap Example

Let’s create an example using malloc(3) and realloc(3p) that gives good insight. Here is how our example will play out:

Timer example in systemd

Sat, 09 Jul 2022 00:00:00 +0000

Timer example in systemd

We can use systemd instead of cron timers in Unix. Using systemd timers is easy and provides numerous benefits over using Cron which are documented below. These timers more are complicated than cron, however, they are much more powerful.

Creating a `systemd` timer

Let us look at the logical hierarchy of a timer on systemd and the files associated before we look at an example.

Logical Requirements

To create a systemd timer the following are required:

Differences between lists in Unix

Mon, 27 Jun 2022 00:00:00 +0000

Differences between lists in Unix

Unix comm command - Difference between two lists

Using the comm(1) command in Unix we can output the difference between sorted lists. The comm command takes two sorted files as inputs and will output lines unique in FILE1, lines unique in FILE2 and lines common to FILE1 and FILE2. The comm command requires the files to be sorted.

Examples

 comm -12 file1 file2
 Print only lines present in both file1 and file2.

 comm -3 file1 file2
 Print lines in file1 not in file2, and vice versa.

Example with Files

Let’s see an example by creating two lists in file a.txt and file b.txt. Then use the comm command to output what is only in a.txt, in b.txt and in both a.txt and b.txt - the union between them.

Unix Command for Epoch Time

Sun, 06 Feb 2022 00:00:00 +0000

Unix Command for Epoch Time

This post provides some examples using the Unix date(1) command to interact with Epoch Time. We commonly need to print or parse the Epoch.

Print the Epoch

Using the Unix date command we can print the Epoch. We need to use the %s format.

$ date +%s
1644156059

The %s format is seconds since the Epoch (1970-01-01 00:00 UTC). The Epoch is always in the UTC or Zulu time zone.

USB 3D Sound on the Raspberry Pi

Fri, 01 Jan 2021 00:00:00 +0000

USB 3D Sound on the Raspberry Pi

I recently bought the USB 3D Sound Card to play around with some C ALSA Programming on the Raspberry Pi. It worked without a problem and wanted to jot down some notes in this post on it’s specifications and how to use it. I didn’t have to make any modifications to the Raspberry Pi at all and sound worked right out of the box. I ran this on a Raspberry Pi B+ on Raspian.

Unix Process Tutorial

Sat, 28 Nov 2020 00:00:00 +0000

Unix Process Tutorial

In this blog post I’ll go over a quick tutorial on Unix Processes. We’ll start with some theory and then go into some real world examples. It’s meant to be quick, to the point, and provide the necessary information to have a good understanding about how Unix Processes are designed.

SSH’ing into a Unix Machine

Let’s start with a common scenario. This scenario is to simply ssh into another Unix machine. From a Unix process perspective what happens?

Retry in Bash

Mon, 21 Sep 2020 00:00:00 +0000

Retry in Bash

In this blog post we’ll have two examples in bash that will retry in the following ways:

Retry a command N times before failing
Retry command for X minutes before failing

In both of these examples we’ll use true and false as examples to test the script. These would need to be changed for your use case.

Retry a Command X Times

A script to run a command N times. We will fail after 3 attempts, and delay for 10 seconds between attempts.

Systemd: Timers

Tue, 30 Jun 2020 00:00:00 +0000

Systemd: Timers

In the fourth post on how to create a service in systemd we will create a timer that will run every minute sending messages to our example foo service over a socket. We’re essentially creating a cron job socket client.

Project Creation of an autotools project and start/stop a daemon service.
Journalling/Syslog to enable logging, notify of state changes and accept reloads.
Socket creation of a Unix Domain Socket for other processes to send messages to our service through remote procedure calls.

What the example will do?

We already have a Unix Domain Socket created by systemd that is listening on /var/log/foo.socket. We also have a service called foo.service that is listening to this socket . When messages are received on the socket the service will log them to the journal and syslog.

Systemd: A Service and a Socket

Fri, 26 Jun 2020 00:00:00 +0000

Systemd: A Service and a Socket

This is the third post on how to create a service in systemd. See the first post to create a autotools project and start/stop a daemon service. Or the second post to enable logging, notify of state changes and accept reloads. In this post we’ll create a Unix Domain Socket so that other processes can send messages to our service through remote procedure calls.

The socket we will use in this post is a Unix Socket with domain AF_UNIX of type SOCK_DGRAM. This type of socket is well suited for example purposes and is also ideal for remote procedure call applications. It is limited to communication on the same host, not over the internet. Unix Domain sockets are always reliable and are guaranteed to be delivered in order without duplication. Since this example uses a datagram socket we can receive the full message on each reception. Typically, the size of a datagram socket is under 2048 bytes see socket(7) and look for the SO_SNDBUF option for more details on maximum datagram size.

Status, Reloading and Journalling in Systemd

Tue, 23 Jun 2020 00:00:00 +0000

Status, Reloading and Journalling in Systemd

This is the second post on how to create a service in systemd. In the first post we created an autotools project and were able to start, stop and get basic status on the service. In this post we’ll build on the autotools project to add logging to syslog and journald. We will also give an example on how we could reload the service with a systemctl reload foo. We will also use the sd_notify(3) API to notify the service manager the status of what the service is doing.

Systemd and Autotools

Mon, 22 Jun 2020 00:00:00 +0000

Systemd and Autotools

This post starts a tutorial series on how to create a systemd service in Linux. This first post is likely the hardest part as we create an autotools project with the source code and systemd unit files. Having this setup in autotools saves significant time installing unit files that systemd requires, building our C code, installing, and uninstalling our program. If you can get through this first post the following posts should come easily.

I/O Multiplexing in Unix

Sun, 26 Apr 2020 00:00:00 +0000

I/O Multiplexing in Unix

In Unix a process performs I/O on a single file descriptor at a time. When we talk about I/O on a file think read() and write() calls which are blocking. To get around blocking on a I/O call there are a number of ways. You can imagine how blocking on a single I/O call could bring the performance of a program to it’s knees. In this post we’ll quickly compare non-blocking I/O Models. Namely, select(), poll(), Signal Driven I/O, and epoll().

Unix Command Line Tool Timeline

Wed, 25 Mar 2020 00:00:00 +0000

A compact timeline of Unix command line tools, ordered by the earliest release or broad adoption date.

These entries focus on the core command line utilities that power Unix shells and classic Unix-style workflows. The list intentionally avoids large application suites, window managers, or programming languages used only incidentally as commands.

Timeline

Year	Tool	Notes
1971	`sh`	The original Bourne shell, the foundation of Unix command-line scripting.
1971	`cat`, `cp`, `mv`, `rm`, `ls`, `mkdir`, `rmdir`, `pwd`, `chmod`	Core file and directory utilities from early Unix releases.
1973	`grep`	Pattern searching on files, one of the most widely used Unix commands.
1974	`awk`	A powerful text-processing tool designed for Unix data manipulation.
1974	`sed`	Stream editing for filtering and transforming text.
1974	`find`	Recursive file discovery and selection.
1975	`sort`, `uniq`, `cut`, `paste`, `head`, `tail`	Classic text processing commands for pipeline workflows.
1975	`tar`	Archive creation and extraction, important for backups and distribution.
1979	`make`	Build automation for compiling and managing source trees.

Scope

This page is meant to capture the history of Unix-style shell tools, not every executable that happens to offer a CLI. If a tool feels like an integral Unix utility, it belongs here; if it is primarily a larger application or programming environment, it does not.

Daemon Example in C

Thu, 05 Mar 2020 08:24:36 +0000

Daemon Example in C

In this post we’ll look at daemon creation and demonstrate how to programmatically create daemons. We’ll go into the SysV recommendation of “double forking” for daemon creation.

What are Daemons?

Daemons are long running processes that run in the background with no controlling terminal - tty. Use cases for daemons are when the program needs to be available at all times and managed by the scheduler. Popular examples are nginx, postfix, httpd, sshd, cron, inetd. Many of these end in “d”, which is a convention for the name of a daemon. Daemons typically do not have the ability to write to stdout, or stderr, and have no means to connect to stdin because they have no controlling terminal. Having no controlling terminal is a big deal in daemons. If they could connect to a controlling terminal they could be used in nefarious ways. We guarantee the daemon cannot take on a controlling terminal by doing the so-called “double fork”. Usually, output from the daemon is done by writing to log files. These logs are typically written in the in /var/log directory. Input to a daemon, when necessary, is typically through sockets and signals. Let’s write an example daemon in C that runs on Unix.

Unix Virtual Network Device Types

Fri, 07 Feb 2020 00:00:00 +0000

Unix Virtual Network Device Types

This blog post attempts to list and describe the majority of Virtual Network Device types in Unix. Networking with Linux is a hot topic especially in the IoT and Datacenter areas. Read on and you’ll see a large array of Device Types found in Unix.

Background on Network Configuration in Unix

The iproute2 framework replaces ifconfig as well as many other commands. The command family ip link deals with network device configuration in Unix. We can use iproute2 to add, change, and delete network configurations. When adding a network using iproute2 we can do a ip link add name NAME type TYPE where TYPE is one of the many Linux Networking Device Types.

Creating IP Tunnels in Linux with iproute2

Tue, 04 Feb 2020 07:40:31 +0000

Creating IP Tunnels in Linux with iproute2

In this post we’ll use iproute2 to create tunnels between two unix hosts. This will be the simplest form of an IP-in-IP tunnel.

How the Tunnels will work

In this example we have two Unix hosts with IP addresses 10.255.254.96 and 10.255.254.196 on a 10.255.254.0/24 network and a gateway address of 10.255.254.2. These two hosts are on the same network. For this tunnel example we are going to add dummy interfaces to each of our hosts and assign addresses 192.168.2.111 and 192.168.2.222 addresses to each of them respectively. Once the tunnels are up we will be able to send a ping from the first tunnel to the second tunnel as if the hosts are on the same network. This is known as encapsulation, since inside an IPv4 packet with a source of 10.255.254.96 and destination of 10.255.254.196 we have another IPv4 packet encapsulated inside with a source of 192.168.2.111 and destination of 192.168.2.222.

pbcopy/pbpaste in WSL

Tue, 30 Jul 2019 00:00:00 +0000

pbcopy/pbpaste in WSL

If you’re looking to replicate the pbcopy and pbpaste commands in WSL then you’ve found the right place! In this post we will make some simple shell scripts to replicate pbcopy and pbpaste in OS X. If you’re accustomed to pbcopy/pbpaste in Mac OS X, it’s hard to not have the equivalent in WSL. One spoiler alert the commands just are not as fast as they are in OS X.

Processing Emails with a script in Postfix

Sat, 27 Apr 2019 07:56:00 +0000

Updated Feb 7, 2020

After thinking about this post more, it’s true you can process emails with a script in Postfix. The post provides the right answer to the wrong question!! You can certainly process emails with Postfix but it’s hard, brittle, reduces performance and really isn’t recommended. What is better is to have a client program that reads the mail and does what you desire. This client can be put on a cron job, triggered or run however you want. The client is very flexible and so many different programming languages support them. This gives a lot of power to mark emails read, manage folders, etc … It’s just a cleaner, better, easier to use and test architecture than having a script in Postfix. So I’ll leave this up, but, I wouldn’t use it unless there is some reason I cannot think of how it would be better.

Unix on Lloyd Rochester's Geek Blog

Understanding Process Execution

Understanding Process Execution

Remote Process Execution

UART Theory and Programming on the Raspberry Pi

UART Theory and Programming on the Raspberry Pi

Understanding memory reallocation with realloc

Understanding memory reallocation with realloc

Table of Contents

Heap Example

Timer example in systemd

Timer example in systemd

Creating a systemd timer

Logical Requirements

Differences between lists in Unix

Differences between lists in Unix

Unix comm command - Difference between two lists

Examples

Example with Files

Unix Command for Epoch Time

Unix Command for Epoch Time

Table of Contents

Print the Epoch

USB 3D Sound on the Raspberry Pi

USB 3D Sound on the Raspberry Pi

Unix Process Tutorial

Unix Process Tutorial

SSH’ing into a Unix Machine

Retry in Bash

Retry in Bash

Retry a Command X Times

Systemd: Timers

Systemd: Timers

What the example will do?

Systemd: A Service and a Socket

Systemd: A Service and a Socket

Status, Reloading and Journalling in Systemd

Status, Reloading and Journalling in Systemd

Systemd and Autotools

Systemd and Autotools

I/O Multiplexing in Unix

I/O Multiplexing in Unix

Unix Command Line Tool Timeline

Timeline

Scope

Daemon Example in C

Daemon Example in C

Table of Contents

What are Daemons?

Unix Virtual Network Device Types

Unix Virtual Network Device Types

Background on Network Configuration in Unix

Creating IP Tunnels in Linux with iproute2

Creating IP Tunnels in Linux with iproute2

How the Tunnels will work

pbcopy/pbpaste in WSL

pbcopy/pbpaste in WSL

Processing Emails with a script in Postfix

Creating a `systemd` timer