Batch Processing with SOA
Although typically thought of as an artifact of legacy computing, batch processes remain vital to today’s real-time enterprises. Behind the real time systems that power the real time enterprise, such as customer order fulfillment, account management, supply chain scheduling and optimization, or financial trading systems, are regularly-updated back office business systems. Today, batch processes remain essential for one key reason: it is simply not efficient to regenerate a complete forecast or business plan every time the business processes a single event such as an incoming customer order. Real time enterprises do require systems that can support dynamic processes; however, it is best to reserve that capacity for aspects of data or processes for the most volatile high-velocity markets.
Nonetheless, while the need for batch processes hasn’t changed, the nature of batch processing today has certainly evolved. For instance, while scheduled batch processes remain relevant for time-related processes such as end-of-period reporting, real time enterprises may require more flexibility for adapting to temporary or permanent market fluctuations, or to planned or unplanned changes in underlying IT infrastructure. Companies are faced with new requirements for managing compliance with increasingly stringent regulatory mandates. This may dictate the need for policy-driven workflows with the capability for dynamically triggering batch processes when specific scenarios arise. For instance, a Sarbanes Oxley violation might trigger a rollback and new round of updating core financial systems outside of the normal schedule.
Service-Oriented Architecture (SOA) presents enterprises with the opportunity to expose information and processes as self-contained Services that can communicate and interoperate with each other in a standard, loosely coupled fashion. Although the common impression is that Services expose business processes, data processes, or application functionality, they are also well-suited for exposing the very processes that drive batch-oriented workloads. SOA enables the business to build flexible compositions of Services that implement either business or IT processes in a loosely coupled manner, which has important ramifications for IT service delivery, and the batch processes that are part of it.
The Evolution of Batch Processing
Over the years, batch technology has evolved from script-based automation to rules or policy-driven workload automation. IT has long sought to automate the running of batch jobs, initially with tools that automatically chained them together in a schedule. As batch windows shrunk, IT organizations embraced job management technologies enabling them to optimize their use of limited resources over shorter timeframes. Workload automation is the culmination of innovations in job automation, adapting it to the needs of the real time enterprise with policy-driven, configurable workflows. Compared to traditional job automation methods, configurable workflows are far more flexible and maintainable.
Yet, the objective has changed. Instead of optimizing time windows or resource consumption in the data center, today the mission of batch processing has broadened its support of the business. In the real time enterprise, today’s batch operation is morphing from a static, often standalone process to a dynamic component of an application or composite business process. The goals could include accelerating responsiveness, improving corporate transparency, and enabling IT to adopt a more business-centric and compliance-driven approach to managing its own operations.
The key to achieving these goals is consistency and agility. With traditional, hard-coded or manual approaches, ensuring uniformity of batch processes was often a hit or miss proposition. Batch, and its successor workload automation, has evolved over time, with the latest innovations adding policy or rules-driven workflows that transform batch jobs into repeatable workflows.
Ultimately, SOA provides the architectural underpinnings that bring this vision to fruition: By invoking batch processes as a Service, they become ingrained components of the composable business processes that power the real time enterprise. For instance, via Service enablement, workload automation can become extensions of applications ranging from supply chain management to IT service desk. Long a stepchild of data center operations, workload automation transforms into a first class citizen of the real time enterprise.
Embedding workload automation as a Service could also encourage cultural changes that bring together IT operations, software development, and the business. Typically, batch jobs are either scheduled or “thrown over the wall” from the business to operations. When batch workloads or workflows exposed as Services become extensions of the application, developers who design or configure the application can now factor in the demand on IT infrastructure by specifying the logic or rules under which enterprise systems can request batch Services. Workload automation is the key to making the batch window a first class citizen of the real time enterprise, which dictates expanding the role of workflow automation from an IT to a business-driven tool.
Consequently, the goal of workload automation changes as well. Instead of IT optimizing its own data center update workflows, applications or business processes are now in the driver’s seat, triggering workloads when specific business events or scenarios occur. This requires a business process-focused workflow engine for job automation, and a business rules engine for enabling policy-based workflow management. In a business-driven workflow, requests from applications, database stored procedures, or business processes trigger workloads (which may consist of one or more batch jobs) as a result of explicit business rules or policies.
For instance, when order activity for a particular product SKU occurs, the following rules could be used to automatically trigger new workloads:
- If a handful of unexpected orders arrive, alter shipment schedules, but do not change supply chain plans.
- If there is an unexpected run on the product, generate a new demand forecast.
- If the run causes actual or potential product shortages, regenerate inventory, procurement, distribution, and demand plans.
Under these scenarios, workloads no longer strictly focus on optimizing database updates in the data center. Instead, they have become intrinsic components of demand-driven supply chain business processes. Batch processes would be exposed as Services that you might want to access in real time. The idea of a batch is that the process is pre-defined (so, a batch is not a flexible process), but it would be exposed as a Service that might be consumed in a flexible process. In traditional workloads, a batch might be scheduled as part of a daily activity, but a Service-oriented batch could be executed on demand. From a Service-oriented perspective, the Service contract and policy would have to specify how many times a batch can be executed in a single day, how long the batch takes to operate, and any other Service or data dependencies.
The Contribution of SOA to Workload Automation
SOA is all about exposing information and processes as self-contained Services that can communicate and interoperate with each other in a standard, loosely coupled manner, enabling the business to build flexible compositions of Services that implement business processes. Ultimately, SOA enables IT to more effectively align with the business, because it changes the way IT delivers the solutions. The self-contained nature of Services and the standard connectivity empowers IT to rapidly compose solutions by reusing existing Services, resulting in faster time-to benefit compared to traditional ways of developing, modifying, and integrating conventional, monolithic software applications.
Although typically associated with functionality that is exposed from software applications and/or composed business processes, SOA can also expose IT infrastructure processes such as workload automation as Services. Using SOA, business events trigger specific job types. In effect, the job becomes part of a composite, Service-Oriented Business Application. The SOA infrastructure coupled with the standards used in implementing SOA facilitate a true loose coupling where business logic, business, rules, and job specifications can each change without impacting the other components. For instance, the workload automation system could invoke a batch update Service as a result of Business Intelligence system triggering a scheduled Extract-Transform-Load (ETL) operation to a data warehouse. If the user changes their ETL system itself or some of the transformation routines, that change is kept internal to the ETL Service and should not impact the Workload Service.
The ZapThink Take
During the height of the dot.com bubble, the conventional wisdom was that the Internet was supposed to change everything in business. Although the Internet helped enterprises forge new connections to business partners and customers alike, it did not change the fundamentals of conducting business or managing IT systems. Similarly, today’s ascendance of the real time enterprise has not eliminated the need for effective batch processing. By leveraging Service-Oriented Architecture (SOA), enterprises can evolve batch workloads from standalone data center operations to components that are intrinsic to composite, Service-Oriented Business Applications. Where workload automation and batch processing were once arcane tasks relegated deep in the organization, they become critical Services delivered as part of the Enterprise Architecture.