Optimization Techniques for Data-Intensive Decision Flows

Many workflow applications require highly differentiated treatments of inputs. Examples include many electronic commerce and customer case applications, where the customer treatments are highly customized, sometimes to the extent of so called "segment-of-one marketing". An efficient approach to supporting such applications involves incremental decision making, where initial data is gathered, and then iteratively additional data is gathered and/or derived based on the results obtained so far. Current workflow models can support this kind of reasoning, using an acyclic directed graph to guide what data should be gathered/derived for different kinds of inputs. This paper introduces and empirically analyzes a variety of optimization strategies for use in workflow systems that involve data-intensive incremental decision making. A primary focus ois on the use of parallelism and eagerness (aka speculative execution) to reduce response time without exceeding the capacity of available resources.