Moab Workload Manager Administrator's Guide

version 5.2

Copyright © 1999-2008 Cluster Resources, Inc All Rights Reserved
Distribution of this document for commercial purposes in either
hard or soft copy form is strictly prohibited
without prior written consent from Cluster Resources, Inc.

Overview

Moab Workload Manager is a highly advanced scheduling and management system designed for clusters, grids, and on-demand/utility computing systems. At a high level, Moab applies site policies and extensive optimizations to orchestrate jobs, services, and other workload across the ideal combination of network, compute, and storage resources. Moab enables true adaptive computing allowing compute resources to be customized to changing needs and failed systems to be automatically fixed or replaced. Moab increases system resource availability, offers extensive cluster diagnostics, delivers powerful QoS/SLA features, and provides rich visualization of cluster performance through advanced statistics, reports, and charts.

Moab works with virtually all major resource management and resource monitoring tools. From hardware monitoring systems like IPMI to system provisioning systems and storage managers, Moab takes advantage of domain expertise to allow these systems to do what they do best, importing their state information and providing them with the information necessary to better do their job. Moab uses its global information to coordinate the activities of both resources and services, which optimizes overall performance in-line with high-level mission objectives.

Table of Contents

• 1.0 Philosophy and Goals of the Moab Workload Manager
  • 1.1 Value of a Batch System
  • 1.2 Philosophy and Goals
  • 1.3 Workload
• 2.0 Installation and Initial Configuration
  • 2.1 Building and Installing Moab
  • 2.2 Initial Moab Configuration
  • 2.3 Initial Moab Testing (Monitor, Interactive, Simulation and Normal Modes)
• 3.0 Scheduler Basics
  • 3.1 Layout of Scheduler Components
  • 3.2 Scheduling Environment and Objects
  • 3.3 Scheduling Iterations and Job Flow
  • 3.4 Configuring the Scheduler
  • 3.5 Credential Overview
• 4.0 Scheduler Commands
  • 4.1 Client Overview
  • 4.2 Monitoring System Status
  • 4.3 Managing Jobs
  • 4.4 Managing Reservations
  • 4.5 Configuring Policies
  • 4.6 End-user Commands
  • 4.7 Miscellaneous Commands
• 5.0 Prioritizing Jobs and Allocating Resources
  • 5.1 Job Prioritization
  • 5.2 Node Allocation
  • 5.3 Node Access
  • 5.4 Node Availability
  • 5.5 Task Distribution
  • 5.6 Virtualization and Resource Provisioning
• 6.0 Managing Fairness - Throttling Policies, Fairshare, and Allocation Management
  • 6.1 Fairness Overview
  • 6.2 Usage Limits/Throttling Policies
  • 6.3 Fairshare
  • 6.4 Charging and Allocation Management
  • 6.5 Internal Charging Facilities
• 7.0 Controlling Resource Access - Reservations, Partitions, and QoS Facilities
  • 7.1 Advance Reservations
  • 7.2 Partitions
  • 7.3 QoS Facilities
• 8.0 Optimizing Scheduling Behavior - Backfill, Node Sets, and Preemption
  • 8.1 Optimization Overview
  • 8.2 Backfill
  • 8.3 Node Sets
  • 8.4 Preemption
• 9.0 Statistics, Accounting, and Profiling
  • 9.1 Scheduler Performance Evaluation Overview
  • 9.2 Accounting - Job and System Statistics
  • 9.3 Testing New Versions and Configurations
  • 9.4 Answering What If? Questions with the Simulator
• 10.0 Managing Shared Resources - Usage Based Limits, Policies, and SMP Issues
  • 10.1 Consumable Resource Handling
  • 10.2 Load Balancing Features
  • 10.3 Resource Usage Tracking
  • 10.4 Resource Usage Limits
  • 10.5 General SMP Issues
• 11.0 General Job Administration
  • 11.1 Deferred Jobs and Job Holds
  • 11.2 Job Priority Management
  • 11.3 Suspend/Resume Handling
  • 11.4 Checkpoint/Restart
  • 11.5 Job Dependencies
  • 11.6 Setting Job Defaults and Per Job Limits
  • 11.7 General Job Policies
  • 11.8 Using a Local Queue
  • 11.9 Job Deadline Support
  • 11.10 Job Templates
• 12.0 General Node Administration
  • 12.1 Node Location (Partitions, Racks, Queues, etc.)
  • 12.2 Node Attributes (Node Features, Speed, etc.)
  • 12.3 Node Specific Policies (MaxJobPerNode, etc.)
  • 12.4 Configuring Node-Locked Consumable Generic Resources (tape drives, node-locked licenses, etc.)
  • 12.5 Managing Shared Cluster Resources (shared filesystems, licenses, etc.)
  • 12.6 Node State Management
• 13.0 Resource Managers and Interfaces
  • 13.1 Resource Manager Overview
  • 13.2 Resource Manager Configuration
  • 13.3 Resource Manager Extensions
  • 13.4 Adding Resource Manager Interfaces
  • 13.5 Managing Resources Directly with the Native Interface
  • 13.6 Utilizing Multiple Resource Managers
  • 13.7 License Management
  • 13.8 Provisioning Managers
  • 13.9 Network Management
  • 13.10 Integrating with Hardware Managers
  • 13.11 Enabling Resource Manager Translation
• 14.0 Troubleshooting and System Maintenance
  • 14.1 Internal Diagnostics
  • 14.2 Logging Facilities
  • 14.3 Object Messages
  • 14.4 Notifying Administrators of Failures and Critical Events
  • 14.5 Issues with Client Commands
  • 14.6 Tracking System Failures
  • 14.7 Problems with Individual Jobs
  • 14.8 Diagnostic Scripts
• 15.0 Improving User Effectiveness
  • 15.1 User Feedback Loops
  • 15.2 User Level Statistics
  • 15.3 Enhancing Wallclock Limit Estimates
  • 15.4 Providing Resource Availability Information
  • 15.5 Job Start Time Estimates
  • 15.6 Collecting Performance Information on Individual Jobs
• 16.0 Cluster Analysis, Testing, and Simulation
  • 16.1 Evaluating New Releases and Policies
  • 16.2 Testing New Middleware
  • 16.3 Simulation Overview
• 17.0 Moab Workload Manager for Grids
  • 17.1 Grid Basics
  • 17.2 Grid Configuration
  • 17.3 Centralized Grid Management
  • 17.4 Source-Destination Grid Management
  • 17.5 Localized Grid Management
  • 17.6 Resource Control and Access
  • 17.7 Workload Submission and Control
  • 17.8 Reservations in the Grid
  • 17.9 Grid Usage Policies
  • 17.10 Grid Scheduling Policies
  • 17.11 Grid Credential Management
  • 17.12 Data Staging
  • 17.13 Grid Data Management
  • 17.14 Accounting and Allocation Management
  • 17.15 Grid Security
  • 17.16 Configuring a Grid with Globus
  • 17.17 Grid Information Services
  • 17.18 Grid Diagnostics and Validation
  • 17.19 Overflow Grid Case Study
  • 17.20 Data Staging Case Study
• 18.0 Utility Computing with Moab (Hosting on Demand)
  • 18.1 Utility Computing Overview
  • 18.2 Resource Monitoring in a Utility Computing Hosting Center
  • 18.3 Resource Allocation and Environment Management
  • 18.4 Provisioning Resources and Managing Dynamic Security
  • 18.5 Service Level Agreements and QoS Guarantees
  • 18.6 Tight Integration with Customer Resources
  • 18.7 Enabling Manual Resource Access Requests
  • 18.8 Automating Resource Access Requests
  • 18.9 Standby Resources
  • 18.10 Accounting, Costing, and Automated Billing
  • 18.11 Utility Computing Example: Setting up a Test Center in 1 Hour or Less
• 19.0 Object Triggers
  • 19.1 Trigger Creation
  • 19.2 Trigger Management
  • 19.3 Trigger Components
  • 19.4 Trigger Types
  • 19.5 Trigger Variables
  • 19.6 Trigger Examples
• 20.0 Virtual Private Clusters
  • 20.1 Virtual Private Clusters Overview
• 21.0 Miscellaneous
  • 21.1 User Feedback
  • 21.2 Enabling High Availability Features
  • 21.3 Using the Application Scheduling Library
  • 21.4 Dynamic and Malleable Jobs
  • 21.5 Identity Managers
  • 21.6 Information Services for the Enterprise and Grid
• Appendices
  • Appendix A: Case Studies
  • Appendix B: Extension Interface
  • Appendix C: Adding New Algorithms
  • Appendix D: Adjusting Default Limits
  • Appendix E: Security Configuration
  • Appendix F: Parameters Overview
  • Appendix G: Commands Overview
  • Appendix H: Interfacing to Moab
  • Appendix I: Considerations for Large Clusters
  • Appendix J: Adding Moab as a Service
  • Appendix K: Migrating from Maui 3.2
  • Appendix O: Resource Manager Integration Overview
  • Appendix Q: Moab in the Data Center