Further Reading

If you missed the first two parts of this series, start there — they introduce the concepts of topology strings and binding basics that this article builds on:

Compute Nodes with Heterogeneous Topology in Gridware Cluster Scheduler

Understanding Binding in Gridware Cluster Scheduler

Why order and range matter

Imagine your cluster as a row of houses, each with several rooms. Jobs are like guests arriving to stay overnight.

Without a plan, guests will pick rooms at random. Some might share walls and benefit from proximity; others might end up in different houses entirely.

Binding order and range options give the scheduler a plan — a way to decide how the “houses” and “rooms” are filled:

• -bsort defines which house or room type to consider first.
• -bstart sets where the first guest can enter.
• -bstop defines where the group must stop filling.

It’s a simple idea — but it’s the foundation for consistent, balanced, and sometimes energy-efficient scheduling. This ensures that jobs don’t overlap or scatter unpredictably — they follow a defined parking pattern.

1. Sorting with -bsort

The -bsort option controls the order in which available CPU units are considered. It uses the same letters as topology strings (S, C, E, N, X, Y) but interprets them by utilization:

• Uppercase letters (S, C, N…) mean start with free resources.
• Lowercase letters (s, c, n…) mean start with already-used resources.

This simple mechanism lets administrators describe sophisticated fill-up patterns:

• -bsort "SC" — Fill unutilized sockets and cores first.
• -bsort "sC" — Reuse partially filled sockets before opening a new one.
• -bsort "nSyC" — (GCS only) Use utilized NUMA nodes and 3rd Level Caches before starting opening new ones but empty sockets and cores are preferred within NUMA nodes and core groups.

If no sort order is defined, the scheduler uses the hardware’s natural order as reported by HWLOC. This order is the default behavior in most competitive workload management systems and ensures predictable placement when the WLM lacks a possibility to specify preferences.

2. Defining the range with -bstart and -bstop

After sorting, the scheduler knows which binding units exist and in what order they should be considered. The next question is: how much of that ordered list should be used? That’s what -bstart and -bstop control.

You can think of the sorted topology as a long sequence of nodes — a string that lists sockets, cores, or cache domains in the order defined by -bsort.

With -bstart and -bstop, you mark a window inside that string to show where binding is allowed.

Start here → [================] ← Stop here

The brackets represent the usable part of the topology. Binding can happen only inside this bracketed region; everything outside it is ignored. This is especially useful when you want to reserve or exclude certain parts of a host — for example, to keep the first socket free for system services or to group jobs within a specific NUMA region.

Consider a dual-socket, quad-core host represented as:

SCCccSCCCC

(lowercase c marks partially utilized cores)

• -bstart S -bstop s → the scheduler begins binding at the first unutilized socket (S) and stops before the first partially used socket (s), effectively limiting the job to the clean half of the machine. (SCCcc[SCCCC])
• -bstart s -bstop S → Reversing the range starts binding at the first used socket (s) and continues until the next free socket (S) ([SCCcc]SCCCC)

You can imagine this as managing a parking lot: -bsort arranges the parking spaces by preference, and -bstart/-bstop draw the lines around the section that is currently open for parking. Jobs will always stay within that section, preventing overlap and ensuring predictable placement.

3. Real-world significance

At first glance, -bsort, -bstart, and -bstop may seem like low-level tweaks. In reality, they are the building blocks of enterprise scheduling strategy.

These options influence how load, heat, and power consumption are distributed across hardware, and they provide the basis for higher-level policies such as:

Energy-Aware Scheduling – fill one socket or die before activating another.

Thermal & Power-Density Balancing – distribute workloads evenly across nodes.

License- or Cost-Aware Placement – schedule on specific CPU’s first.

Cache & Memory-Bandwidth Optimization – keep related processes close to shared caches or NUMA regions.

In GCS, JSV (Job Submission Verifier) scripts can dynamically adjust these settings to enforce global policies across the cluster.

4. Looking ahead

This post closes the “mechanics” phase of the binding series. In the next article, we’ll apply everything we’ve learned to real enterprise use cases — combining sorting, filtering, and range selection to implement:

• Energy-optimized cluster configurations
• Thermal-balanced compute fabrics
• Cost- and license-aware job scheduling

What seems like three small options today will soon become the core tools for advanced resource optimization.

For many years, qmon was the graphical management tool for Grid Engine and its derivatives. Maintaining this legacy Motif-based application had become increasingly difficult — both technically and from a usability perspective.
With Qontrol, we are starting fresh: a modern, responsive, browser-based interface designed to support administrators and users of large HPC environments.

Qontrol provides direct insight into cluster status, jobs, queues, and resource usage — all presented through a clean and flexible UI that can adapt to a wide range of display environments.
It is built to integrate seamlessly with existing Gridware Cluster Scheduler installations, requiring no additional client software.

The initial release focuses on monitoring and visualization, but we are actively working on interactive management capabilities and customizable dashboards for upcoming versions.

You can find the full introduction and feature overview in our main post:
👉 Introducing Qontrol – A Modern Web UI for Gridware Cluster Scheduler

Stay tuned — this marks the beginning of a new generation of tools built around Gridware Cluster Scheduler.

Why Binding Matters

In high-performance computing (HPC), resource binding defines how processes or threads are mapped to specific CPU resources. Effective binding ensures predictable performance by preventing multiple jobs from competing for the same core, cache, or memory subsystem. It also enhances bandwidth utilization for attached devices such as network interfaces, InfiniBand adapters, and GPUs by maintaining locality between compute tasks and their associated hardware resources..

Gridware Cluster Scheduler treats binding as a first-class resource. Unlike traditional schedulers where binding was merely a hint, in Gridware Cluster Scheduler it is a hard requirement — a job will only start once the requested binding can be fulfilled.

From Slots to Binding

If you’ve read the previous post, you already know about the slot concept — where each slot represents a unit of computational capacity on a node. Here’s the full analogy that makes the difference between slots and binding concrete:

• Slots = seats on an airplane. A compute node has a fixed number of slots, just as a plane has a fixed number of seats. Each sequential job and each task of a parallel job needs one slot, like each passenger needs one seat.
• Binding = weight-balanced placement and freight. Binding determines where the job runs within the node. In the airplane analogy, that’s like assigning specific rows/sections and placing freight in defined compartments to maintain balance. Similarly, binding pins tasks to threads, cores, sockets, dies (L3), or NUMA nodes so they benefit from nearby caches and memory and don’t interfere with other workloads.

In short: slots define how many, binding defines where — the placement that preserves locality and stability.

Binding Units and Amounts

Binding behavior is primarily controlled with the -bunit and -bamount parameters during job submission.

Define the Binding Level with -bunit <unit>

Each unit level corresponds to a layer in the hardware hierarchy.

Specify the Number of Units with -bamount <number>

The binding amount defines how many binding units should be assigned per slot (or per host).

qsub -pe mpi_8 16 -bunit C -bamount 2 ...

This job requests 16 slots across two hosts. Each slot binds to two power cores, ideal for tasks starting two lightweight threads (or processes).

If the threads are tightly coupled, thread binding can be more suitable:

qsub -pe mpi_8 16 -bunit T -bamount 2 ...

Binding threads instead of cores can enhance total cluster throughput by minimizing stalls from system calls, cache misses, and network delays, and by improving cache locality (especially for producer–consumer pairs). While each job may take up to twice as long to complete, the increased parallelism—running twice as many jobs simultaneously—often results in a net performance improvement of 5–10%.

Chiplet or die binding can be especially beneficial on modern CPUs where groups of cores share a common L3 cache. By aligning tasks to those chiplets, cache locality is preserved and cross-die memory traffic is minimized.

qsub ... -btype host -bunit X -bamount 1

This command binds each job or task to all cores that share the same L3 cache. It ensures that the job exclusively uses that cache domain and its attached resources (e.g. GPU or I/O devices), preventing other jobs from interfering. Also this can result in a performance benefit even if the job might not use all cores of the die.

Binding Types: Slot vs. Host

Binding can be applied per slot (as we saw in previous examples) or per host using the -btype parameter.

• Slot-based binding (default):
  Each slot gets its own binding. This maximizes flexibility and is ideal for mixed workloads.
• Host-based binding:
  Binding is applied collectively for all slots on a host, ensuring consistent placement but reducing flexibility.

qsub -pe mpi_8 16 -btype host -bunit X -bamount 1 ...

Here, the job get 16 slots (8 per host) and Gridware Cluster Scheduler binds each group to one die (L3-cache). This host-wide approach minimizes fragmentation and improves cache locality.

Once a job (or advance reservation) is scheduled, its actual binding can be inspected with qstat (or qrstat)

qstat -j <job_id>   # or  qrstat -ar <ar_id>
...
binding:               bamount=16,binstance=set,bstrategy=pack,btype=host,bunit=X
exec_binding_list 1:   host1=NSxccccccccXCCCCCCCC,host2=NSxccccccccXCCCCCCCC

The first line shows the binding request; the second lists the binding actually applied per host (lower case letters in the topology string). In this example, all cores below the first L3 cache of the first socket were used.

Binding Filters

Sometimes certain cores or sockets should be left free — for example, one core per host reserved for system tasks. Binding filters, defined with -bfilter, make this possible.

A filter uses a topology string where lowercase letters mark excluded units.

Example:

qsub -bfilter ScCCCScCCC ...

Here, the first core of each socket is masked and will not be used for binding. All other cores remain available.

Administrators can also define global filters by keyword:

qconf -sconf | grep binding_params
binding_params ... filter=first_core

Global and job-specific filters are additive, and both restrictions apply simultaneously.

Packed Binding (and What Comes Next)

The packed binding strategy is the default in Gridware Cluster Scheduler.
It assigns available hardware units sequentially from left to right within a node’s topology string, ensuring that each host is filled efficiently while maintaining cache and NUMA locality.

Packed binding automatically groups tasks on nearby cores and within shared cache domains to reduce latency and memory contention.
If a host does not have enough free units to satisfy a job’s binding request, the scheduler simply skips that host.

What has been written so far is available in Open Cluster Scheduler (OCS) and Gridware Cluster Scheduler (GCS).

However, Gridware Cluster Scheduler introduces extended binding control. Packed binding can be refined through additional options — -bsort, -bstart, and -bstop — which let you influence the order and region of unit selection.

These advanced strategies are available only in Gridware Cluster Scheduler and will be discussed in detail in the next blog post.

🚀 Stay connected!
Follow me on X (Twitter) or join HPC-Gridware on LinkedIn and X (Twitter) for the latest release announcements, expert tips, and in-depth technical insights from our team.

🔧 Try it today: nightly builds featuring the latest OCS and GCS enhancements discussed in this post are now available from HPC-Gridware.

Why Topology Awareness Matters

In modern high-performance computing (HPC), CPU cores within a single socket may differ in clock frequency, power characteristics, or cache layout. Meanwhile, memory hierarchies—NUMA nodes, multi-level caches, and chiplets—add new layers of complexity.

To schedule jobs efficiently, a cluster manager must understand and exploit this hardware topology.
Gridware Cluster Scheduler 9.1.0 introduces expanded binding and topology-awareness features to maximize performance and ensure predictable resource placement.

Three Hardware Topologies from NVIDIA, AMD, and Intel

To demonstrate the scheduler’s new capabilities, the following sections show real-world topology examples from NVIDIA, AMD, and Intel hardware.

NVIDIA DGX Spark

The NVIDIA DGX Spark—notable for its presentation by Jensen Huang to Elon Musk at SpaceX—uses a heterogeneous ARM architecture optimized for AI/ML workloads.
The system features 20 ARM cores organized into five performance tiers, each with unique efficiency and frequency characteristics:

CPU-Type #4: efficiency=4, cpuset=0x00080000
  FrequencyMaxMHz = 4004
  LinuxCapacity   = 1024
CPU-Type #3: efficiency=3, cpuset=0x00078000
  FrequencyMaxMHz = 3978
  LinuxCapacity   = 1017
CPU-Type #2: efficiency=2, cpuset=0x000003e0
  FrequencyMaxMHz = 3900
  LinuxCapacity   = 997
CPU-Type #1: efficiency=1, cpuset=0x00007c00
  FrequencyMaxMHz = 2860
  LinuxCapacity   = 731
CPU-Type #0: efficiency=0, cpuset=0x0000001f
  FrequencyMaxMHz = 2808
  LinuxCapacity   = 718

Using Intel’s terminology, this architecture could be viewed as 10 Power cores and 10 Efficiency cores
(10 × ARM Cortex-X925 + 10 × ARM Cortex-A725). Each core has private L1/L2 caches, and groups of 10 share an L3 cache.

> loadcheck -cb | grep Topology
Topology (GCS): NSXEEEEECCCCCXEEEEECCCCC

Gridware Cluster Scheduler uses topology strings to represent such layouts.
Here: N = NUMA node, S = socket, X = L3 cache, E = Efficiency core, and C = Power core.

Intel i9-14900HX

While the Intel i9-14900HX isn’t typical for HPC clusters, it’s an ideal case study for hybrid core architectures.

> loadcheck -cb | grep Topology
Topology (GCS): NSXCTTCTTCTTCTTCTTCTTCTTCTTYEEEEYEEEEYEEEEYEEEE

• Power cores (C): Dual-threaded (T), each with its own L2 cache.
• Efficiency cores (E): Single-threaded, grouped by four per L2 cache (Y).
• NUMA node (N) and socket (S): Encompass both core types and a shared L3 cache (X).

AMD EPYC Zen5

The AMD EPYC Zen5 series (e.g., AMD-Epyc-Zen5-c4d-highmem-384) represents a chiplet-based homogeneous design.
Each core provides two hardware threads, and the L3 cache structure (X) maps directly to chiplets/dies.

> loadcheck -cb | grep Topology
Topology (GCS): NSXCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  XCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  ... (repeated chiplet layout per socket)
                NSXCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  XCTTCTTCTTCTTCTTCTTCTTCTT XCTTCTTCTTCTTCTTCTTCTTCTT
                  ... (repeated chiplet layout per socket)

Each socket (S) corresponds to one NUMA node (N), while every core has a private L2 cache.

Handling Heterogeneous Topologies in Gridware Cluster Scheduler

Efficient scheduling means assigning tasks to the most suitable hardware.
If a parallel job spans both slow and fast cores, the slowest becomes a bottleneck. Similarly, crossing NUMA or cache boundaries increases latency.

Gridware Cluster Scheduler 9.1 introduces fine-grained binding control, allowing binding to:

• Sockets
• Cores
• Threads
• NUMA nodes
• Chiplets/Dies (cache domains)

This ensures optimal locality and predictable performance, even on hybrid or asymmetric systems.

Chiplet/Die Binding Example

qsub -pe mpi 15 -btype host -bamount 2 -bunit X ...

This example requests 15 MPI tasks, all running on a single host. Using -btype host, binding is applied relative to the host topology. With -bamount 2 -bunit X, each job portion binds to two chiplets/dies, ensuring that cache boundaries are respected and minimizing cross-die interference.

💡 In this setup, the job uses 15 out of 16 available cores. The scheduler keeps the remaining core idle to prevent contention.

Summary

With version 9.1.0, Gridware Cluster Scheduler becomes fully topology-aware, bridging the gap between modern heterogeneous hardware and intelligent workload scheduling.
By supporting multiple binding types (core, socket, thread, NUMA, chiplet/die), it ensures efficient resource utilization and predictable performance across diverse compute nodes.

We are currently in the QA phase of this release and welcome user feedback on these new features.
They are already included in our nightly builds for testing, and beta releases will be available soon.
Download Gridware Cluster Scheduler

Stay tuned with HPC-Gridware for updates — we’ll share more insights, examples, and best practices as we approach the official release.

Follow me at X/Twitter or follow us at HPC-Gridware (LinkedIn, X/Twitter) for release announcements, tips, and technical insights.

OCS and GCS v9.0.8 are now available. As usual, the packages can be downloaded from the HPC-Gridware download page, and the source code is available on the Cluster Scheduler GitHub project page.

The list of fixed issues mentioned in the Release Notes can be found here:

Improvement

CS-739 qstat -j output should contain job state, start time, queue name, and host names

Task

CS-1407 Add SUSE SLES 15 support in support matrix of release notes

CS-1440 Add qtelemetry licenses to GCS 3rdparty licenses directory

CS-1470 do memory testing on V90_BRANCH for the 9.0.8 release

Sub-task

CS-1394 Add start_time of array jobs tasks to qstat -j

CS-1395 Cleanup of job states and show states also in qstat -j output

CS-1396 Show granted host information in qstat -j output

CS-1404 Show granted queues in qstat -j output

CS-1410 Show priority in qstat -j output even if it is the base priority

Bug

CS-671 qrsh truncates the command line and/or output at 927 characters

CS-1019 sge_execd logs errors when running tightly integrated parallel jobs

CS-1270 Installation script clears screen in case of an error which make issues harder to debug

CS-1381 qacct complains “error: ignoring invalid entry in line 436” for accounting records with huge command line entry

CS-1386 man page for sge_share_mon is missing

CS-1403 sge_ckpt man-page is in wrong section (1 instead of 5)

CS-1422 endless loop in protocol between sge_qmaster and sge_execd in certain job failure situations

CS-1424 qmod -sj on own job fails on submit only host

CS-1429 sge_qmaster can segfault on qdel -f

CS-1434 clearing error state of a job leads to event callback error logging in qmaster messages file

CS-1435 rescheduling of jobs requires manager rights, documented is “manager or operator rights”

CS-1436 qmod man pages says it requires manager or operator privileges to rerun a job, but a job owner may rerun his own jobs as well

CS-1451 option -out of examples/jobsbin//work is broken

CS-1476 Go DRMAA does not set JoinFiles() correctly

CS-1477 In Go DRMAA TransferFiles() does not set all values

Please let me or the HPC-Gridware team know if you have any questions.