Package org.opennms.netmgt.rrd

Class QueuingRrdStrategy

  • All Implemented Interfaces:
    Runnable, RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>
    public class QueuingRrdStrategy
extends Object
implements RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>, Runnable
    Provides queuing implementation of RrdStrategy. In order to provide a more scalable collector. We created a queuing RrdStrategy that enabled the system to amortize the high cost of opening an round robin database across multiple updates. This RrdStrategy implementation enqueues the create and update operations on a per file basis and maintains a set of threads that process enqueued work file by file. If the I/O system can keep up with the collection threads while performing only a single update per file then eventually all the data is processed and the threads sleep until there is more work to do. If the I/O system is initially slower than than the collection threads then work will enqueue here and the write threads will get behind. As this happens each file will eventually have more than a single update enqueued and therefore the number of updates pushed thru the system will increase because more then one will be output per 'open' Eventually, the I/O system and the collection system will balance out. When this happens all data will be collected but will not be output to the rrd files until the next time the file is processed by the write threads. As another performance improving strategy. The queue distinguishes between files with significant vs insignificant updates. Files with only insignificant updates are put at the lowest priority and are only written when the highest priority updates have been written This implementation delegates all the actual writing to another RrdStrategy implementation. System properties effecting the operation: org.opennms.rrd.queuing.writethreads: (default 2) The number of rrd write threads that process the queue org.opennms.rrd.queuing.queueCreates: (default false) indicates whether rrd file creates should be queued or processed synchronously org.opennms.rrd.queuing.maxInsigUpdateSeconds: (default 0) the number of seconds over which all files with significant updates only should be promoted onto the significant less. This is to ensure they don't stay unprocessed forever. Zero means not promotion. org.opennms.rrd.queuing.modulus: (default 10000) the number of updates the get enqueued between statistics output org.opennms.rrd.queuing.category: (default "queued") the log routing prefix to place the statistics output in TODO: Promote files when ZeroUpdate operations can't be merged. This may be a collection miss which we want to push thru. It should also help with memory. TODO: Set an upper bound on enqueued operations TODO: Provide an event that will write data for a particular file... Say right before we try to graph it.
    Version:
    $Id: $
    Author:
    ranger

    • Constructor Detail

      • QueuingRrdStrategy

        public QueuingRrdStrategy​(RrdStrategy<Object,​Object> delegate)

        Constructor for QueuingRrdStrategy.

        Parameters:
        delegate - a RrdStrategy object.

    • Method Detail

      • getConfigurationProperties

        public Properties getConfigurationProperties()

        getConfigurationProperties

        Returns:
        a Properties object.

      • getWriteThreads

        public int getWriteThreads()

        getWriteThreads

        Returns:
        a int.

      • setWriteThreads

        public void setWriteThreads​(int writeThreads)

        setWriteThreads

        Parameters:
        writeThreads - a int.

      • queueCreates

        public boolean queueCreates()

        queueCreates

        Returns:
        a boolean.

      • setQueueCreates

        public void setQueueCreates​(boolean queueCreates)

        setQueueCreates

        Parameters:
        queueCreates - a boolean.

      • prioritizeSignificantUpdates

        public boolean prioritizeSignificantUpdates()

        prioritizeSignificantUpdates

        Returns:
        a boolean.

      • setPrioritizeSignificantUpdates

        public void setPrioritizeSignificantUpdates​(boolean prioritizeSignificantUpdates)

        setPrioritizeSignificantUpdates

        Parameters:
        prioritizeSignificantUpdates - a boolean.

      • getInSigHighWaterMark

        public long getInSigHighWaterMark()

        getInSigHighWaterMark

        Returns:
        a long.

      • setInSigHighWaterMark

        public void setInSigHighWaterMark​(long inSigHighWaterMark)

        setInSigHighWaterMark

        Parameters:
        inSigHighWaterMark - a long.

      • getSigHighWaterMark

        public long getSigHighWaterMark()

        getSigHighWaterMark

        Returns:
        a long.

      • setSigHighWaterMark

        public void setSigHighWaterMark​(long sigHighWaterMark)

        setSigHighWaterMark

        Parameters:
        sigHighWaterMark - a long.

      • getQueueHighWaterMark

        public long getQueueHighWaterMark()

        getQueueHighWaterMark

        Returns:
        a long.

      • setQueueHighWaterMark

        public void setQueueHighWaterMark​(long queueHighWaterMark)

        setQueueHighWaterMark

        Parameters:
        queueHighWaterMark - a long.

      • getModulus

        public long getModulus()

        getModulus

        Returns:
        a long.

      • setModulus

        public void setModulus​(long modulus)

        setModulus

        Parameters:
        modulus - a long.

      • getCategory

        public String getCategory()

        getCategory

        Returns:
        a String object.

      • setCategory

        public void setCategory​(String category)

        setCategory

        Parameters:
        category - a String object.

      • getMaxInsigUpdateSeconds

        public long getMaxInsigUpdateSeconds()

        getMaxInsigUpdateSeconds

        Returns:
        a long.

      • setMaxInsigUpdateSeconds

        public void setMaxInsigUpdateSeconds​(long maxInsigUpdateSeconds)

        setMaxInsigUpdateSeconds

        Parameters:
        maxInsigUpdateSeconds - a long.

      • getWriteThreadSleepTime

        public long getWriteThreadSleepTime()

        getWriteThreadSleepTime

        Returns:
        a long.

      • setWriteThreadSleepTime

        public void setWriteThreadSleepTime​(long writeThreadSleepTime)

        setWriteThreadSleepTime

        Parameters:
        writeThreadSleepTime - a long.

      • getWriteThreadExitDelay

        public long getWriteThreadExitDelay()

        getWriteThreadExitDelay

        Returns:
        a long.

      • setWriteThreadExitDelay

        public void setWriteThreadExitDelay​(long writeThreadExitDelay)

        setWriteThreadExitDelay

        Parameters:
        writeThreadExitDelay - a long.

      • createDefinition

        public QueuingRrdStrategy.CreateOperation createDefinition​(String creator,
                                                           String directory,
                                                           String rrdName,
                                                           int step,
                                                           List<RrdDataSource> dataSources,
                                                           List<String> rraList)
                                                    throws Exception
        Create a round robin database definition from the supplied parameters. This definition is used in the createFile call to create the actual file.
        Specified by:
        createDefinition in interface RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>
        Parameters:
        creator - - A string representing who is creating this file for use in log msgs
        directory - - The directory to create the file in
        rrdName - - The name to use for the round robin database
        step - - the step for the database
        dataSources - - the data sources to use for round robin database
        rraList - - a List of the round robin archives to create in the database. defines after which time the data is condensed to a defined lower step
        Returns:
        an object representing the definition of an round robin database. Can be null if the database exists already.
        Throws:
        Exception - If an error occurs while creating the definition

      • openFile

        public String openFile​(String fileName)
                throws Exception
        Opens the round robin database with the supplied name. It is assumed the name refers to a round robin database appropriate for this strategy implementation
        Specified by:
        openFile in interface RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>
        Parameters:
        fileName - the name of the associated rrd file
        Returns:
        an open rrd reference that can by used in calls to updateFile and closeFile
        Throws:
        Exception - if an error occurs opening the file

      • fetchLastValueInRange

        public Double fetchLastValueInRange​(String rrdFile,
                                    String ds,
                                    int interval,
                                    int range)
                             throws NumberFormatException,
                                    RrdException
        Fetches the last value from the round robin database with the given name within a time range. The interval passed in should be the interval associated with the round robin database. The range should be the amount of "lag" acceptable for an update to be considered valid. Range must be a multiple of the RRD interval.
        Specified by:
        fetchLastValueInRange in interface RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>
        Parameters:
        rrdFile - a name the represents a round robin database
        ds - a name the represents a data source to be used
        interval - a step interval of the round robin database
        range - an acceptable range for which the last value will be returned
        Returns:
        The last value as a Double (if the last value didn't exist returns a Double.NaN)
        Throws:
        NumberFormatException - if any.
        RrdException - if any.

      • createGraph

        public InputStream createGraph​(String command,
                               File workDir)
                        throws IOException,
                               RrdException
        Creates an InputStream representing the bytes of a graph created from round robin data. It accepts an rrdtool graph command. The underlying implementation converts this command to a format appropriate for it .
        Specified by:
        createGraph in interface RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>
        Parameters:
        command - the command needed to create the graph
        workDir - the directory that all referenced files are relative to
        Returns:
        an input stream representing the bytes of a graph image as a PNG file
        Throws:
        IOException - if an IOError occurs
        RrdException - if an RRD error occurs

      • run

        public void run()

        run

        Specified by:
        run in interface Runnable

      • createGraphReturnDetails

        public RrdGraphDetails createGraphReturnDetails​(String command,
                                                File workDir)
                                         throws IOException,
                                                RrdException
        Creates an RrdGraphDetails object representing the graph created from round robin data. It accepts an rrdtool graph command. The underlying implementation converts this command to a format appropriate for it .
        Specified by:
        createGraphReturnDetails in interface RrdStrategy<QueuingRrdStrategy.CreateOperation,​String>
        Parameters:
        command - the command needed to create the graph
        workDir - the directory that all referenced files are relative to
        Returns:
        details for the graph including an InputStream, any PRINTed lines, and graph dimensions.
        Throws:
        IOException - if an IOError occurs
        RrdException - if an RRD error occurs

      • getTotalOperationsPending

        public long getTotalOperationsPending()

        getTotalOperationsPending

        Returns:
        a long.

      • setTotalOperationsPending

        public void setTotalOperationsPending​(long totalOperationsPending)

        setTotalOperationsPending

        Parameters:
        totalOperationsPending - a long.

      • getCreatesCompleted

        public long getCreatesCompleted()

        getCreatesCompleted

        Returns:
        a long.

      • setCreatesCompleted

        public void setCreatesCompleted​(long createsCompleted)

        setCreatesCompleted

        Parameters:
        createsCompleted - a long.

      • getUpdatesCompleted

        public long getUpdatesCompleted()

        getUpdatesCompleted

        Returns:
        a long.

      • setUpdatesCompleted

        public void setUpdatesCompleted​(long updatesCompleted)

        setUpdatesCompleted

        Parameters:
        updatesCompleted - a long.

      • getErrors

        public long getErrors()

        getErrors

        Returns:
        a long.

      • setErrors

        public void setErrors​(long errors)

        setErrors

        Parameters:
        errors - a long.

      • getPromotionCount

        public long getPromotionCount()

        getPromotionCount

        Returns:
        a long.

      • setPromotionCount

        public void setPromotionCount​(long promotionCount)

        setPromotionCount

        Parameters:
        promotionCount - a long.

      • getSignificantOpsEnqueued

        public long getSignificantOpsEnqueued()

        getSignificantOpsEnqueued

        Returns:
        a long.

      • setSignificantOpsEnqueued

        public void setSignificantOpsEnqueued​(long significantOpsEnqueued)

        setSignificantOpsEnqueued

        Parameters:
        significantOpsEnqueued - a long.

      • getSignificantOpsDequeued

        public long getSignificantOpsDequeued()

        getSignificantOpsDequeued

        Returns:
        a long.

      • setSignificantOpsDequeued

        public void setSignificantOpsDequeued​(long significantOpsDequeued)

        setSignificantOpsDequeued

        Parameters:
        significantOpsDequeued - a long.

      • getEnqueuedOperations

        public long getEnqueuedOperations()

        getEnqueuedOperations

        Returns:
        a long.

      • setEnqueuedOperations

        public void setEnqueuedOperations​(long enqueuedOperations)

        setEnqueuedOperations

        Parameters:
        enqueuedOperations - a long.

      • getDequeuedOperations

        public long getDequeuedOperations()

        getDequeuedOperations

        Returns:
        a long.

      • setDequeuedOperations

        public void setDequeuedOperations​(long dequeuedOperations)

        setDequeuedOperations

        Parameters:
        dequeuedOperations - a long.

      • getDequeuedItems

        public long getDequeuedItems()

        getDequeuedItems

        Returns:
        a long.

      • setDequeuedItems

        public void setDequeuedItems​(long dequeuedItems)

        setDequeuedItems

        Parameters:
        dequeuedItems - a long.

      • getSignificantOpsCompleted

        public long getSignificantOpsCompleted()

        getSignificantOpsCompleted

        Returns:
        a long.

      • setSignificantOpsCompleted

        public void setSignificantOpsCompleted​(long significantOpsCompleted)

        setSignificantOpsCompleted

        Parameters:
        significantOpsCompleted - a long.

      • getStartTime

        public long getStartTime()

        getStartTime

        Returns:
        a long.

      • setStartTime

        public void setStartTime​(long updateStart)

        setStartTime

        Parameters:
        updateStart - a long.