And then it happened.

The first bite was a surprising moment — almost ridiculously intense.

The sweet creaminess of Nutella melts into the warm, aromatic spices of the bread. The coriander seed (a citrusy, warm spice — not cilantro) adds a bright, unexpected lift that harmonizes beautifully with the chocolate-hazelnut sweetness.

It was as if my brain thought simultaneously:

“I know this!”
and
“What on earth is this — and why is it so good?!”

After four decades of breakfast routine, it felt like rediscovering something familiar. An upgrade to a memory. A tiny revolution on a slice of bread.

Why It Works (Especially If You’ve Never Had German Spice Bread)

• Spice bread isn’t overly sweet.
  It’s closer to a rustic loaf with warm spices than to American-style sweet breads.
• Coriander seed is the secret weapon.
  It has a lemony, floral warmth — completely different from fresh cilantro.
  In spice bread, it adds depth that plays surprisingly well with Nutella.
• Nutella gains complexity.
  Its sweetness and fat carry the spice notes, turning a childhood favorite into something almost gourmet.

Conclusion

Spice bread with Nutella is no longer just an experiment — it’s here to stay.
And maybe it shows that shaking up old habits now and then is worth it.

If you’ve been eating the same breakfast for years:
Try something unexpected. Your personal “wow” moment might be waiting right next to the toaster.

(A Note on Palm-Oil-Free Options)

If you love the idea of this combination but prefer a palm-oil-free chocolate-hazelnut spread, there are some excellent alternatives out there. Brands like Rigoni di Asiago Nocciolata (Organic), Nutvia, Vego Chocolate Spread, or Milford Choco Nussa (Palm-oil-free version) offer rich, creamy spreads without palm oil.

Further Reading

If you missed the first two parts of this series, start there — they introduce the concepts of topology strings and binding basics that this article builds on:

Compute Nodes with Heterogeneous Topology in Gridware Cluster Scheduler

Understanding Binding in Gridware Cluster Scheduler

Why order and range matter

Imagine your cluster as a row of houses, each with several rooms. Jobs are like guests arriving to stay overnight.

Without a plan, guests will pick rooms at random. Some might share walls and benefit from proximity; others might end up in different houses entirely.

Binding order and range options give the scheduler a plan — a way to decide how the “houses” and “rooms” are filled:

• -bsort defines which house or room type to consider first.
• -bstart sets where the first guest can enter.
• -bstop defines where the group must stop filling.

It’s a simple idea — but it’s the foundation for consistent, balanced, and sometimes energy-efficient scheduling. This ensures that jobs don’t overlap or scatter unpredictably — they follow a defined parking pattern.

1. Sorting with -bsort

The -bsort option controls the order in which available CPU units are considered. It uses the same letters as topology strings (S, C, E, N, X, Y) but interprets them by utilization:

• Uppercase letters (S, C, N…) mean start with free resources.
• Lowercase letters (s, c, n…) mean start with already-used resources.

This simple mechanism lets administrators describe sophisticated fill-up patterns:

• -bsort "SC" — Fill unutilized sockets and cores first.
• -bsort "sC" — Reuse partially filled sockets before opening a new one.
• -bsort "nSyC" — (GCS only) Use utilized NUMA nodes and 3rd Level Caches before starting opening new ones but empty sockets and cores are preferred within NUMA nodes and core groups.

If no sort order is defined, the scheduler uses the hardware’s natural order as reported by HWLOC. This order is the default behavior in most competitive workload management systems and ensures predictable placement when the WLM lacks a possibility to specify preferences.

2. Defining the range with -bstart and -bstop

After sorting, the scheduler knows which binding units exist and in what order they should be considered. The next question is: how much of that ordered list should be used? That’s what -bstart and -bstop control.

You can think of the sorted topology as a long sequence of nodes — a string that lists sockets, cores, or cache domains in the order defined by -bsort.

With -bstart and -bstop, you mark a window inside that string to show where binding is allowed.

Start here → [================] ← Stop here

The brackets represent the usable part of the topology. Binding can happen only inside this bracketed region; everything outside it is ignored. This is especially useful when you want to reserve or exclude certain parts of a host — for example, to keep the first socket free for system services or to group jobs within a specific NUMA region.

Consider a dual-socket, quad-core host represented as:

SCCccSCCCC

(lowercase c marks partially utilized cores)

• -bstart S -bstop s → the scheduler begins binding at the first unutilized socket (S) and stops before the first partially used socket (s), effectively limiting the job to the clean half of the machine. (SCCcc[SCCCC])
• -bstart s -bstop S → Reversing the range starts binding at the first used socket (s) and continues until the next free socket (S) ([SCCcc]SCCCC)

You can imagine this as managing a parking lot: -bsort arranges the parking spaces by preference, and -bstart/-bstop draw the lines around the section that is currently open for parking. Jobs will always stay within that section, preventing overlap and ensuring predictable placement.

3. Real-world significance

At first glance, -bsort, -bstart, and -bstop may seem like low-level tweaks. In reality, they are the building blocks of enterprise scheduling strategy.

These options influence how load, heat, and power consumption are distributed across hardware, and they provide the basis for higher-level policies such as:

Energy-Aware Scheduling – fill one socket or die before activating another.

Thermal & Power-Density Balancing – distribute workloads evenly across nodes.

License- or Cost-Aware Placement – schedule on specific CPU’s first.

Cache & Memory-Bandwidth Optimization – keep related processes close to shared caches or NUMA regions.

In GCS, JSV (Job Submission Verifier) scripts can dynamically adjust these settings to enforce global policies across the cluster.

4. Looking ahead

This post closes the “mechanics” phase of the binding series. In the next article, we’ll apply everything we’ve learned to real enterprise use cases — combining sorting, filtering, and range selection to implement:

• Energy-optimized cluster configurations
• Thermal-balanced compute fabrics
• Cost- and license-aware job scheduling

What seems like three small options today will soon become the core tools for advanced resource optimization.

For many years, qmon was the graphical management tool for Grid Engine and its derivatives. Maintaining this legacy Motif-based application had become increasingly difficult — both technically and from a usability perspective.
With Qontrol, we are starting fresh: a modern, responsive, browser-based interface designed to support administrators and users of large HPC environments.

Qontrol provides direct insight into cluster status, jobs, queues, and resource usage — all presented through a clean and flexible UI that can adapt to a wide range of display environments.
It is built to integrate seamlessly with existing Gridware Cluster Scheduler installations, requiring no additional client software.

The initial release focuses on monitoring and visualization, but we are actively working on interactive management capabilities and customizable dashboards for upcoming versions.

You can find the full introduction and feature overview in our main post:
👉 Introducing Qontrol – A Modern Web UI for Gridware Cluster Scheduler

Stay tuned — this marks the beginning of a new generation of tools built around Gridware Cluster Scheduler.

1. Patterns Are Just Algorithms in Yarn

Knitting patterns are basically human-readable pseudocode:

“K2, P2, repeat until despair.”

It’s the same logic as:

for i in range(desperation):
    knit(2)
    purl(2)

Both rely on precise syntax. One wrong parenthesis or missed yarn-over, and suddenly your scarf has the same structural integrity as early JavaScript.

And of course, every pattern says “just repeat rows 1–12 until desired length”, which is the same energy as a junior dev being told to “just scale it horizontally.”

2. Debugging: Frogging Your Way to Sanity

In knitting, “frogging” means ripping your work back to fix a mistake. (Because you literally go “rip it, rip it.”)

In programming, that’s called refactoring, reverting, or sometimes “throwing the whole repo away and pretending it never happened.”

Both involve denial, bargaining, and at least one existential sigh.

If you’ve ever thought, “maybe no one will notice that mistake”, congrats — you’ve reached Stage 4 of the debugging grief cycle.

3. Version Control: The Knitter’s Project Bag

Every knitter has at least three half-finished projects living in different tote bags, quietly judging them from the corner.

Programmers call these side projects.
Both groups say things like:

“I’ll finish it one day.”

They won’t.

Also, neither group understands what’s actually inside the bag/repo anymore. Opening it is like running git log on your own shame.

4. Libraries, Frameworks, and Yarn Dependencies

Knitting:

“I just downloaded this new lace pattern library!”

Programming:

“I just installed 47 npm packages to print ‘Hello, World.’”

Both involve dependencies that you don’t fully understand but somehow trust your life with.
The only difference is that knitters’ libraries don’t randomly break when someone unpublishes left-pad.

5. The Compile Moment

Knitting’s version of hitting “run” is binding off.
That’s the glorious instant when you hold up your finished sweater and realize… it’s two sizes too small.

It’s the textile equivalent of a program that works on your machine.

Both situations involve long pauses, quiet swearing, and a vow that next time, you’ll “read the pattern more carefully.” You won’t.

6. Community & Open Source

Knitters have Ravelry, an online pattern-sharing platform with 9 million users.
Programmers have GitHub, an online pattern-sharing platform with 100 million users and roughly the same number of arguments about best practices.

Both communities are powered by generosity, obsession, and strong opinions about tools.
Circular needles vs. straight needles? Vim vs. VS Code? Same energy.

7. The Joy (and Futility) of Creation

At the end of the day, both crafts deliver the same hit of dopamine.
You start with nothing — just a string (of yarn or text) — and end up with something that exists because you made it.

It’s imperfect, a little weird, but undeniably yours.

And when someone asks, “Wow, you made that yourself?”
You smile, nod, and think:

“Yes. And I spent six hours debugging it because I missed a comma.”

Epilogue

So the next time someone tells you knitting is old-fashioned, remind them it’s basically low-level parallel processing. And when someone says programming is too abstract, remind them: it’s just digital crochet with extra parentheses.

Either way, we’re all just trying to get our loops to run without tangling.

Why Binding Matters

In high-performance computing (HPC), resource binding defines how processes or threads are mapped to specific CPU resources. Effective binding ensures predictable performance by preventing multiple jobs from competing for the same core, cache, or memory subsystem. It also enhances bandwidth utilization for attached devices such as network interfaces, InfiniBand adapters, and GPUs by maintaining locality between compute tasks and their associated hardware resources..

Gridware Cluster Scheduler treats binding as a first-class resource. Unlike traditional schedulers where binding was merely a hint, in Gridware Cluster Scheduler it is a hard requirement — a job will only start once the requested binding can be fulfilled.

From Slots to Binding

If you’ve read the previous post, you already know about the slot concept — where each slot represents a unit of computational capacity on a node. Here’s the full analogy that makes the difference between slots and binding concrete:

• Slots = seats on an airplane. A compute node has a fixed number of slots, just as a plane has a fixed number of seats. Each sequential job and each task of a parallel job needs one slot, like each passenger needs one seat.
• Binding = weight-balanced placement and freight. Binding determines where the job runs within the node. In the airplane analogy, that’s like assigning specific rows/sections and placing freight in defined compartments to maintain balance. Similarly, binding pins tasks to threads, cores, sockets, dies (L3), or NUMA nodes so they benefit from nearby caches and memory and don’t interfere with other workloads.

In short: slots define how many, binding defines where — the placement that preserves locality and stability.

Binding Units and Amounts

Binding behavior is primarily controlled with the -bunit and -bamount parameters during job submission.

Define the Binding Level with -bunit <unit>

Each unit level corresponds to a layer in the hardware hierarchy.

Specify the Number of Units with -bamount <number>

The binding amount defines how many binding units should be assigned per slot (or per host).

qsub -pe mpi_8 16 -bunit C -bamount 2 ...

This job requests 16 slots across two hosts. Each slot binds to two power cores, ideal for tasks starting two lightweight threads (or processes).

If the threads are tightly coupled, thread binding can be more suitable:

qsub -pe mpi_8 16 -bunit T -bamount 2 ...

Binding threads instead of cores can enhance total cluster throughput by minimizing stalls from system calls, cache misses, and network delays, and by improving cache locality (especially for producer–consumer pairs). While each job may take up to twice as long to complete, the increased parallelism—running twice as many jobs simultaneously—often results in a net performance improvement of 5–10%.

Chiplet or die binding can be especially beneficial on modern CPUs where groups of cores share a common L3 cache. By aligning tasks to those chiplets, cache locality is preserved and cross-die memory traffic is minimized.

qsub ... -btype host -bunit X -bamount 1

This command binds each job or task to all cores that share the same L3 cache. It ensures that the job exclusively uses that cache domain and its attached resources (e.g. GPU or I/O devices), preventing other jobs from interfering. Also this can result in a performance benefit even if the job might not use all cores of the die.

Binding Types: Slot vs. Host

Binding can be applied per slot (as we saw in previous examples) or per host using the -btype parameter.

• Slot-based binding (default):
  Each slot gets its own binding. This maximizes flexibility and is ideal for mixed workloads.
• Host-based binding:
  Binding is applied collectively for all slots on a host, ensuring consistent placement but reducing flexibility.

qsub -pe mpi_8 16 -btype host -bunit X -bamount 1 ...

Here, the job get 16 slots (8 per host) and Gridware Cluster Scheduler binds each group to one die (L3-cache). This host-wide approach minimizes fragmentation and improves cache locality.

Once a job (or advance reservation) is scheduled, its actual binding can be inspected with qstat (or qrstat)

qstat -j <job_id>   # or  qrstat -ar <ar_id>
...
binding:               bamount=16,binstance=set,bstrategy=pack,btype=host,bunit=X
exec_binding_list 1:   host1=NSxccccccccXCCCCCCCC,host2=NSxccccccccXCCCCCCCC

The first line shows the binding request; the second lists the binding actually applied per host (lower case letters in the topology string). In this example, all cores below the first L3 cache of the first socket were used.

Binding Filters

Sometimes certain cores or sockets should be left free — for example, one core per host reserved for system tasks. Binding filters, defined with -bfilter, make this possible.

A filter uses a topology string where lowercase letters mark excluded units.

Example:

qsub -bfilter ScCCCScCCC ...

Here, the first core of each socket is masked and will not be used for binding. All other cores remain available.

Administrators can also define global filters by keyword:

qconf -sconf | grep binding_params
binding_params ... filter=first_core

Global and job-specific filters are additive, and both restrictions apply simultaneously.

Packed Binding (and What Comes Next)

The packed binding strategy is the default in Gridware Cluster Scheduler.
It assigns available hardware units sequentially from left to right within a node’s topology string, ensuring that each host is filled efficiently while maintaining cache and NUMA locality.

Packed binding automatically groups tasks on nearby cores and within shared cache domains to reduce latency and memory contention.
If a host does not have enough free units to satisfy a job’s binding request, the scheduler simply skips that host.

What has been written so far is available in Open Cluster Scheduler (OCS) and Gridware Cluster Scheduler (GCS).

However, Gridware Cluster Scheduler introduces extended binding control. Packed binding can be refined through additional options — -bsort, -bstart, and -bstop — which let you influence the order and region of unit selection.

These advanced strategies are available only in Gridware Cluster Scheduler and will be discussed in detail in the next blog post.

🚀 Stay connected!
Follow me on X (Twitter) or join HPC-Gridware on LinkedIn and X (Twitter) for the latest release announcements, expert tips, and in-depth technical insights from our team.

🔧 Try it today: nightly builds featuring the latest OCS and GCS enhancements discussed in this post are now available from HPC-Gridware.

Why Topology Awareness Matters

In modern high-performance computing (HPC), CPU cores within a single socket may differ in clock frequency, power characteristics, or cache layout. Meanwhile, memory hierarchies—NUMA nodes, multi-level caches, and chiplets—add new layers of complexity.

To schedule jobs efficiently, a cluster manager must understand and exploit this hardware topology.
Gridware Cluster Scheduler 9.1.0 introduces expanded binding and topology-awareness features to maximize performance and ensure predictable resource placement.

Three Hardware Topologies from NVIDIA, AMD, and Intel

To demonstrate the scheduler’s new capabilities, the following sections show real-world topology examples from NVIDIA, AMD, and Intel hardware.

NVIDIA DGX Spark

The NVIDIA DGX Spark—notable for its presentation by Jensen Huang to Elon Musk at SpaceX—uses a heterogeneous ARM architecture optimized for AI/ML workloads.
The system features 20 ARM cores organized into five performance tiers, each with unique efficiency and frequency characteristics:

CPU-Type #4: efficiency=4, cpuset=0x00080000
  FrequencyMaxMHz = 4004
  LinuxCapacity   = 1024
CPU-Type #3: efficiency=3, cpuset=0x00078000
  FrequencyMaxMHz = 3978
  LinuxCapacity   = 1017
CPU-Type #2: efficiency=2, cpuset=0x000003e0
  FrequencyMaxMHz = 3900
  LinuxCapacity   = 997
CPU-Type #1: efficiency=1, cpuset=0x00007c00
  FrequencyMaxMHz = 2860
  LinuxCapacity   = 731
CPU-Type #0: efficiency=0, cpuset=0x0000001f
  FrequencyMaxMHz = 2808
  LinuxCapacity   = 718

Using Intel’s terminology, this architecture could be viewed as 10 Power cores and 10 Efficiency cores
(10 × ARM Cortex-X925 + 10 × ARM Cortex-A725). Each core has private L1/L2 caches, and groups of 10 share an L3 cache.

> loadcheck -cb | grep Topology
Topology (GCS): NSXEEEEECCCCCXEEEEECCCCC

Gridware Cluster Scheduler uses topology strings to represent such layouts.
Here: N = NUMA node, S = socket, X = L3 cache, E = Efficiency core, and C = Power core.

Intel i9-14900HX

While the Intel i9-14900HX isn’t typical for HPC clusters, it’s an ideal case study for hybrid core architectures.

> loadcheck -cb | grep Topology
Topology (GCS): NSXCTTCTTCTTCTTCTTCTTCTTCTTYEEEEYEEEEYEEEEYEEEE

• Power cores (C): Dual-threaded (T), each with its own L2 cache.
• Efficiency cores (E): Single-threaded, grouped by four per L2 cache (Y).
• NUMA node (N) and socket (S): Encompass both core types and a shared L3 cache (X).

AMD EPYC Zen5

The AMD EPYC Zen5 series (e.g., AMD-Epyc-Zen5-c4d-highmem-384) represents a chiplet-based homogeneous design.
Each core provides two hardware threads, and the L3 cache structure (X) maps directly to chiplets/dies.

> loadcheck -cb | grep Topology
Topology (GCS): NSXCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  XCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  ... (repeated chiplet layout per socket)
                NSXCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  XCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  ... (repeated chiplet layout per socket)

Each socket (S) corresponds to one NUMA node (N), while every core has a private L2 cache.

Handling Heterogeneous Topologies in Gridware Cluster Scheduler

Efficient scheduling means assigning tasks to the most suitable hardware.
If a parallel job spans both slow and fast cores, the slowest becomes a bottleneck. Similarly, crossing NUMA or cache boundaries increases latency.

Gridware Cluster Scheduler 9.1 introduces fine-grained binding control, allowing binding to:

• Sockets
• Cores
• Threads
• NUMA nodes
• Chiplets/Dies (cache domains)

This ensures optimal locality and predictable performance, even on hybrid or asymmetric systems.

Chiplet/Die Binding Example

qsub -pe mpi 15 -btype host -bamount 2 -bunit X ...

This example requests 15 MPI tasks, all running on a single host. Using -btype host, binding is applied relative to the host topology. With -bamount 2 -bunit X, each job portion binds to two chiplets/dies, ensuring that cache boundaries are respected and minimizing cross-die interference.

💡 In this setup, the job uses 15 out of 16 available cores. The scheduler keeps the remaining core idle to prevent contention.

Summary

With version 9.1.0, Gridware Cluster Scheduler becomes fully topology-aware, bridging the gap between modern heterogeneous hardware and intelligent workload scheduling.
By supporting multiple binding types (core, socket, thread, NUMA, chiplet/die), it ensures efficient resource utilization and predictable performance across diverse compute nodes.

We are currently in the QA phase of this release and welcome user feedback on these new features.
They are already included in our nightly builds for testing, and beta releases will be available soon.
Download Gridware Cluster Scheduler

Stay tuned with HPC-Gridware for updates — we’ll share more insights, examples, and best practices as we approach the official release.

Follow me at X/Twitter or follow us at HPC-Gridware (LinkedIn, X/Twitter) for release announcements, tips, and technical insights.

Enable Sideloading on the Supernote

• Open Settings on your Supernote and go to Security and Privacy.
• Enable Sideloading by switching the toggle.

Install Google Platform Tools on Your Computer

• Download the SDK Platform Tools.
• Install them on your machine. On Linux, the package includes an application named adb, which is required.

Connect Your Computer and Supernote

• Connect your Supernote to your computer.
• Run the following command to verify the connection

adb devices

Install F-Droid

• F-Droid is an app store for Android devices (including the Supernote).
• Download the fdroid.apk file and scan it with your preferred antivirus tool (e.g., VirusTotal).
• Once verified, upload the APK to your Supernote:

adb install <path>/F-Droid.apk

Install the Aurora Store via F-Droid

• On first launch, F-Droid may take a while to load available apps (requires Wi-Fi).
• Search for Aurora and install the Aurora Store. The Aurora Store is an open-source client for Google Play with a clean design and privacy features.

Install Obsidian from the Aurora Store

Open the Aurora Store and download Obsidian to your Supernote.

Customize Obsidian Settings

• On first launch, create a new vault or sync an existing one.
• For appearance, I recommend installing the Minimal theme under Appearance.
• The theme also has a companion plugin called Minimal Theme Settings.
• Switch the color scheme to E-Ink for an excellent reading and writing experience on the Supernote.

✨ That’s it! You now have Obsidian running on your Supernote, optimized for its e-ink display.

How To Do

Install the zfs-auto-snapshot package

sudo apt install zfs-auto-snapshot

On Ubuntu and Debian, cron scripts are already in place to trigger the script.

Enabling or disabling automatic snapshots for a specific ZFS is done via ZFS properties. This property is inherited by all descendant datasets, so there is no need to set it manually for each one.

sudo zfs set com.sun:auto-snapshot=true zhome/home

You should also specify where you don’t want to see snapshots.

sudo zfs set com.sun:auto-snapshot=false zhome/home/tstusr1
sudo zfs set com.sun:auto-snapshot=false zhome/home/tstusr2

You can enable or disable frequent (every 15 minutes), hourly, daily, weekly or monthly snapshots. They are all enabled by default, so the first snapshot should appear soon.

zfs set com.sun:auto-snapshot:frequent=false zhome/home/tstusr
zfs set com.sun:auto-snapshot:daily=true ...
zfs set com.sun:auto-snapshot:monthly=true ...
...

The following commands are helpful for listing, creating and destroying snapshots:

zfs list -t snapshot

zfs snapshot -r zpool/foo/bar@name

zfs destroy zpool/foo/bar@name

To restore to a certain snapshot, ZFS will roll back all the files and delete all the newer snapshots, so be careful!

zfs rollback zpool/foo/bar@name -r

ZFS itself does not support deleting large numbers of snapshots, but this command comes in handy for that purpose.

sudo zfs list -H -o name -t snapshot zhome/somedir | xargs -n1 zfs destroy

ZFS rollbacks are permanent. They recover everything from the snapshot. If you only want to recover some files or directories, ZFS does not support this directly. However, there is a workaround: Clone the snapshot to a different location and mount the clone. Restore the required files and then delete the clone.

sudo zfs clone zhome/home/ebablick@zfs-auto-snap_frequent-2025-08-25-1845 \
         -o mountpoint=/mnt zhome/ebablick_clone_of_snapshot_2025-08-25-1845

copy /mnt/... /to/...

sudo zfs destroy zhome/ebablick_clone_of_snapshot_2025-08-25-1845 -r

Take a look at the zfs-auto-snapshot(8) manual page. The tool itself offers a few interesting options for executing pre- and post-snapshot commands. The number of snapshots to keep can also be configured via command line arguments, which then need to be adapted for the cron jobs.

OCS and GCS v9.0.8 are now available. As usual, the packages can be downloaded from the HPC-Gridware download page, and the source code is available on the Cluster Scheduler GitHub project page.

The list of fixed issues mentioned in the Release Notes can be found here:

Improvement

CS-739 qstat -j output should contain job state, start time, queue name, and host names

Task

CS-1407 Add SUSE SLES 15 support in support matrix of release notes

CS-1440 Add qtelemetry licenses to GCS 3rdparty licenses directory

CS-1470 do memory testing on V90_BRANCH for the 9.0.8 release

Sub-task

CS-1394 Add start_time of array jobs tasks to qstat -j

CS-1395 Cleanup of job states and show states also in qstat -j output

CS-1396 Show granted host information in qstat -j output

CS-1404 Show granted queues in qstat -j output

CS-1410 Show priority in qstat -j output even if it is the base priority

Bug

CS-671 qrsh truncates the command line and/or output at 927 characters

CS-1019 sge_execd logs errors when running tightly integrated parallel jobs

CS-1270 Installation script clears screen in case of an error which make issues harder to debug

CS-1381 qacct complains “error: ignoring invalid entry in line 436” for accounting records with huge command line entry

CS-1386 man page for sge_share_mon is missing

CS-1403 sge_ckpt man-page is in wrong section (1 instead of 5)

CS-1422 endless loop in protocol between sge_qmaster and sge_execd in certain job failure situations

CS-1424 qmod -sj on own job fails on submit only host

CS-1429 sge_qmaster can segfault on qdel -f

CS-1434 clearing error state of a job leads to event callback error logging in qmaster messages file

CS-1435 rescheduling of jobs requires manager rights, documented is “manager or operator rights”

CS-1436 qmod man pages says it requires manager or operator privileges to rerun a job, but a job owner may rerun his own jobs as well

CS-1451 option -out of examples/jobsbin//work is broken

CS-1476 Go DRMAA does not set JoinFiles() correctly

CS-1477 In Go DRMAA TransferFiles() does not set all values

Please let me or the HPC-Gridware team know if you have any questions.

Setup Steps

Preparing the Device (Repartitioning)

• The NVMe device name can change during a reboot of a Linux system. For example, what was previously /dev/nvme1n1 could now be /dev/nvme0n1, or vice versa. The lsblk -f command is useful for distinguishing between devices that have already been used and those that have just been added.

• wipefs removes old partitions/signatures for a device

sudo wipefs -a /dev/nvme1n1
/dev/nvme1n1: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/nvme1n1: 8 bytes were erased at offset 0xe8e0db5e00 (gpt): 45 46 49 20 50 41 52 54
/dev/nvme1n1: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/nvme1n1: calling ioctl to re-read partition table: Success

• Although there are other tools that can manipulate partition tables, I still use fdisk to create a single large partition on the disk. The fdisk suggestions help to create a partition with the correct alignment, which prevents performance issues or error messages in the subsequent steps. The (g) command creates a new GPT. (p) prints the partition table. (n) triggers the creation of a new partition. (t) allows you to specify a partition type, and (w) makes the data persistent by writing the information to the disk.

sudo fdisk /dev/nvme1n1
Command (m for help): g
Created a new GPT disklabel (GUID: AC0EB50D-4E46-46C7-892D-BEE19E49E76F).

Command (m for help): p
Disk /dev/nvme1n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 990 EVO 1TB                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AC0EB50D-4E46-46C7-892D-BEE19E49E76F

Command (m for help): n
Partition number (1-128, default 1):
Partition number (1-128, default 1): 
First sector (2048-1953525134, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-1953525134, default 1953523711): 
Created a new partition 1 of type 'Linux filesystem' and of size 931,5 GiB.

Command (m for help): p
Disk /dev/nvme1n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 990 EVO 1TB                 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: AC0EB50D-4E46-46C7-892D-BEE19E49E76F
Device         Start        End    Sectors   Size Type
/dev/nvme1n1p1  2048 1953523711 1953521664 931,5G Linux filesystem

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Encryption with LUKS/LUKS2

• The new partition should be LUKS-encrypted. Rather than using device names such as /dev/nvme…, I would like to identify the partition by a UUID. To this end, I will generate a new ID with uuidgen to pass to cryptsetup. The passphrase for the encryption must be specified, and lsblk can be used to verify the result.

uuidgen
168b9568-0540-40b8-b939-882be97eb6bb

cryptsetup luksFormat -q --type luks --sector-size 4096 --cipher aes-xts-plain64 --key-size 256 --uuid 168b9568-0540-40b8-b939-882be97eb6bb --pbkdf argon2i /dev/nvme1n1p1

lsblk -f
...

• The partition can now be opened. I chose the /dev/disk/by-uuid/… path for the device, and the device name will also later be available by the name UUID, prefixed by ‘luks-‘, as with the other partitions initially created by the Debian installer. The ls command on /dev/mapper shows that the partition is available after entering the correct passphrase.

sudo cryptsetup luksOpen --persistent --allow-discards --perf-no_write_workqueue --perf-no_read_workqueue /dev/disk/by-uuid/168b9568-0540-40b8-b939-882be97eb6bb luks-168b9568-0540-40b8-b939-882be97eb6bb
Enter passphrase for /dev/disk/by-uuid/168b9568-0540-40b8-b939-882be97eb6bb:

ls -la /dev/mapper/
...
lrwxrwxrwx  1 root root       7 24. Aug 00:56 luks-168b9568-0540-40b8-b939-882be97eb6bb -> ../dm-2

Opening/Mounting the Device

• Once a partition has been encrypted, it must be activated/opened during the boot process so that it can be used. For the root partition and SWAP of a Debian system, this is done early by GRUB, even before the OS is running. For other partitions, such as our new one, systemd will perform this task. This step must be enabled by installing the necessary package.

sudo apt install systemd-cryptsetup

• Systemd also asks for a passphrase during the boot process, but I want a separate key file that systemd can automatically use to open the encrypted device.
• dd creates a key file containing random data. Access permissions for this file are restricted using the chmod command. The key is then added to the list of authorized keys that can access the device, which requires entering the initial passphrase. Finally, the status is dumped so that it can be verified.

sudo dd if=/dev/urandom of=/crypto_keyfile_home.bin bs=512 count=8

sudo chmod 600 /crypto_keyfile_home.bin

sudo cryptsetup luksAddKey /dev/nvme1n1p1 /crypto_keyfile_home.bin

sudo cryptsetup luksDump /dev/nvme1n1p1

• In order for systemd to recognize the new device, the /etc/crypttab file must be updated with a new entry (in one line).

sudo vi /etc/crypttab
...
luks-168b9568-0540-40b8-b939-882be97eb6bb UUID=168b9568-0540-40b8-b939-882be97eb6bb	/crypto_keyfile_home.bin luks,discard,keyscript=/bin/cat

• After rebooting, the ls command should show that all three encrypted devices are ready.

ls -la /dev/mapper/
....
lrwxrwxrwx  1 root root       7 23. Aug 16:29 luks-011bd881-8ec3-4a12-91df-52e1928539fb -> ../dm-1
lrwxrwxrwx  1 root root       7 24. Aug 00:56 luks-168b9568-0540-40b8-b939-882be97eb6bb -> ../dm-2
lrwxrwxrwx  1 root root       7 23. Aug 16:29 luks-45cabb59-0bb5-4737-b6c9-6007657d7a27 -> ../dm-0

• The device can now be formatted and mounted …

Ubuntu 24.04 is working there quite well. Thanks to TUXEDO, the hardware support if great but one thing what made me cringe was the lack of hibernation support and sporadic wake-ups can be frustrating and dangerous if the notebook should sleep in the bag when you are on the go.

The default Ubuntu 24.04 (and also TUXEDO’s adapted TUXEDO OS) does not setup a SWAP partition of the correct size which makes hibernation impossible. Even if you do this setup yourself you will find out that modes like SuspendThenHibernate will cause the system never to hibernate due to the unnecessary wake-up calls.

The frustration with hibernation was the main reason to give Debian 13 Trixie a try to find out if this distributions also has such issues.

Necessary Steps

I can already summarize that not only was the default installation a success, but hibernation also works perfectly. Minor configuration steps were necessary on Debian 13 Trixie.

• The default installation on NVMe was problem-free. Debian supports LUKS encryption of partitions, including the SWAP partition, which contains the complete extract of the working memory in hibernation mode.
• The SWAP partition automatically has the size of the RAM—more precisely, it is even slightly larger so that the contents of the RAM still fit in the partition despite the LUKS header. A manual hibernation test after installation confirmed that hibernation is working.

sudo systemctl hibernate

• After restarting the notebook, the system wakes up again, asks for the encryption password, boots Linux, and continues where it left off.
• The file /etc/systemd/sleep.conf contains the default settings for systemd, which is responsible for suspend and hibernate. I have adjusted the settings slightly to suit my needs:

sudo vi /etc/systemd/sleep.conf

[Sleep]
AllowSuspend=yes
AllowHibernation=yes
AllowSuspendThenHibernate=yes
AllowHybridSleep=yes
#SuspendState=mem standby freeze
#HibernateMode=platform shutdown
#MemorySleepMode=
HibernateDelaySec=30m
HibernateOnACPower=yes
#SuspendEstimationSec=60min

• In particular, the HibernateDelaySec is set to 30m. In combination with my Gnome settings, this means that Gnome dims the screen after 10 minutes. After 15 minutes, the notebook is suspended for 30 minutes. During this time, touching the touch pad or pressing a key is enough to wake it up, but if you don’t do this, the notebook wakes up on its own after the time has elapsed, only to hibernate.
• The Gnome shutdown menu does not yet include the option to trigger ‘hibernate’, ‘suspend then hibernate’, or ‘hybrid sleep’ from the graphical user interface. However, this can be quickly remedied with a Gnome shell extension called Hibernate Status Button.

• If you’re wondering what hybrid means: In this mode, the memory dump is written before the notebook is suspended and is available when a reboot is required, for example, if the battery runs out during sleep mode.
• When you close the lid of a laptop, Debian Trixie automatically puts the laptop into sleep mode. If you would prefer something else to be triggered instead, this can be changed in /etc/systemd/logind.conf.

sudo vi /etc/systemd/logind.conf

HandleLidSwitch=suspend-then-hibernate
HandleLidSwitchExternalPower=suspend-then-hibernate
...