1. 14 Oct, 2016 1 commit
  2. 11 Oct, 2016 1 commit
    • Wyatt Alt's avatar
      (PDB-3058) version 6 reports incorrectly rejected · f309993c
      Wyatt Alt authored
      This was a schema error introduced with the corrective_change feature.
      The main operational impact was that users upgrading from 3.2.x to 4.2
      with enqueued data would have their commands fail on restart.
      
      This commit corrects the schema to accept v6 reports, and also does a
      bit of test cleanup so we won't have this problem again. Specifically it
      consolidates synthesis of old command versions under the example reports
      namespace and causes test compilation to fail if the synthetic reports
      are invalid.
      f309993c
  3. 10 Oct, 2016 1 commit
  4. 30 Sep, 2016 5 commits
    • Dan Lidral-Porter's avatar
      (maint) Remove suffix check in dlo/metrics-for-dir · 44f759f6
      Dan Lidral-Porter authored
      Remove a redundant check for a ".json" suffix on the filename from
      `puppetdb.command.dlo/metrics-for-dir`, since the metadata parser also
      performs this check, and can handle random strings that aren't the form
      it expects, so there's no need to do this pre-screening.
      44f759f6
    • Dan Lidral-Porter's avatar
      (PDB-2982) Clarify args of embeddable-certname. · d8971ab2
      Dan Lidral-Porter authored
      Clarify that `puppetdb.queue/embeddable-certname` and the two
      functions it calls to determine the certname length all take the
      metadata command name, rather than the command name that the rest of
      PuppetDB uses.
      d8971ab2
    • Dan Lidral-Porter's avatar
      (PDB-2982) Abbrev command names in queue metadata · 6eb6e24d
      Dan Lidral-Porter authored
      Replace `puppetdb.queue/metadata-command-names` with a map called
      `metadata-command->puppetdb-comand` that defines the mapping between
      the command names that can appear in the queue metadata that forms the
      message filenames and the command names used by the rest of PuppetDB.
      The inverse mapping is dynamically constructed under the var
      `puppetdb-command->metadata-command`.
      
      Replace `puppetdb.queue/metadata-str` with `metadata-serializer`, which
      takes a map from PuppetDB command names to the metadata command names
      and returns a function with the same signature as `metadata-str`; viz.
      taking all the queue metadata components and returning a metadata
      string. The only differences are that the returned function looks up the
      command name it's given in the translation map and puts the metadata
      name it finds there in the metadata string, and that if this lookup
      fails then an IllegalArgumentException is thrown.
      
      Bind a metadata serializer with the default translation map of
      `queue/puppetdb-command->metadata-command` to `queue/serialize-metadata`
      
      Change `metadata-parser` to take a translation map rather than
      a sequential collection comand names, have the returned function perform
      the inverse mapping from the metadata command name back to the PuppetDB
      command name, and to add a docstring to `metadata-parser`.
      
      Add `parse-metadata` and `serialize-metadata` vars to the
      `puppetdb.command.dlo` namespace that add an "unknown" -> "unknown"
      entry to both translation maps, since the DLO uses that command name as
      a fallback when the message is so garbled that the command name cannot
      be recovered. Consequently, change all invocations of
      `queue/metadata-str` in the DLO to call `dlo/serialize-metadata`, and
      likewise `queue/metadata-parser` to `dlo/parse-metadata`.
      
      Change `puppetdb.command.dlo/discard-cmdref` to create its own queue
      metadat string and use `stock/entry` directly in order to create
      a stockpile entry for the discarded cmdref, rather than using
      `queue/cmdref->entry`, as the latter creates a metadata string using
      only the default command names & metadata abbreviations, so it will now
      throw an error if the discarded cmdref's command is "unknown".
      
      Update tests to align with the new function names and arguments, as well
      as the abbreviated command names in queue metadata strings.
      6eb6e24d
    • Ryan Senior's avatar
      (PDB-2948) Deprecate ActiveMQ Config · ead9c83c
      Ryan Senior authored
      This commit includes warnings and deprecation notes for the now obsolete
      ActiveMQ config items. The config params no longer apply to PuppetDB
      when running with Stockpile.
      ead9c83c
    • Karel Březina's avatar
  5. 29 Sep, 2016 2 commits
  6. 28 Sep, 2016 1 commit
    • Ryan Senior's avatar
      (PDB-3026) Move the clj -> json byte stream function · 752badf8
      Ryan Senior authored
      This function is useful in extensions, so this commit moves it from
      amq-migration to cheshire. This commit also fixes up an issue with the
      maps being enqueued in the response-mult, causing the extensions tests
      to break.
      752badf8
  7. 27 Sep, 2016 1 commit
    • Dan Lidral-Porter's avatar
      (PDB-2983) Hide rejected execution exceptions. · 283a7657
      Dan Lidral-Porter authored
      Because PuppetDB's command-processing threadpools are unbounded, the
      only reason that they would throw a RejectedExecutionException is
      because they're shutting down or have shut down, which means that
      PuppetDB is shutting down. Commands that are rejected are retried when
      PuppetDB next starts up, so there's no action the user needs to take in
      order to prevent loss of data. Therefore, there's no reason to show the
      user the exceptions that arise when the threadpool throws
      a RejectedExecutionException. To that end, swallow slingshot exceptions
      with kind `:puppetlabs.puppetdb.threadpool/rejected` in the thread
      started by `puppetdb.mq-listener/message-listener-service`'s TK start
      method.
      283a7657
  8. 26 Sep, 2016 7 commits
    • aperiodic's avatar
      (PDB-3039) DLO filenames won't exceed 255 bytes. · 4d1c4b91
      aperiodic authored
      Change the *-filename functions in `puppetlabs.command.dlo` to truncate
      just the metadata portions of the filenames if they would cause the
      filenames to exceed 255 bytes in length. Truncating the metadata portion
      is acceptable since all these filenames contain stockpile IDs, which
      ensures they'll be unique.
      
      Move the `parse-cmd-filename` function, which has to agree with the DLO
      on the format of its discarded message filenames, from puppetdb.queue
      over to the puppetdb.command.dlo namespace, to make it easier to see the
      need for this concordance and to achieve it.
      4d1c4b91
    • Dan Lidral-Porter's avatar
      (PDB-3039) Escape and truncate certnames as needed · 52a91560
      Dan Lidral-Porter authored
      Change `puppetdb.queue/metadata-str` to replace illegal filename
      characters and underscores in the certname it's given, truncate the
      certname if its length would cause the metadata string to exceed 255
      characters, and add the SHA1 hash of the original certname if the
      certname in the metadata string differs in any way (via character
      replacement or truncation) from the original certname.
      
      Add tests of the new sanitization & truncation behavior of
      `puppetdb.queue/metadata-str`.
      52a91560
    • aperiodic's avatar
      (PDB-3039) Add some UTF-8 related utilities. · 20f01480
      aperiodic authored
      Add `utf8-bytes`, `utf8-length`, and `utf8-truncate` functions to the
      utils namespace.
      20f01480
    • Dan Lidral-Porter's avatar
      (maint) Actually quote commands in metadata regex. · d89678e4
      Dan Lidral-Porter authored
      In `puppetdb.queue/metadata-rx`, use \Q and \E to quote the interpolated
      command names in the regex to ignore any regex special characters that
      the command names might contain. Previously, the command names were
      passed through `re-quote-replacement`, which only escapes the special
      characters used in replacement strings: dollar signs and backslashes.
      d89678e4
    • Dan Lidral-Porter's avatar
      (maint) Don't refer queue metadata fns in DLO. · 73d1d685
      Dan Lidral-Porter authored
      Rather than :refer-ing the queue metadata functions, alias the queue
      namespace as `q` and invoke the queue metedata functions using the
      namespace alias.
      73d1d685
    • aperiodic's avatar
      (PDB-3039) Move constants to command.constants · 9fad26d2
      aperiodic authored
      Move some constants from the puppetb.command ns over to
      puppetdb.command.constants, in order to avoid introducing a dependency
      loop in queue.clj in the following commits.
      9fad26d2
    • Ryan Senior's avatar
      (PDB-3026) Moved max-enqueued config to developer · 08db5173
      Ryan Senior authored
      The max-enqueued config is not something users should need, but is more
      useful for developers. This setting sets the maximum number of enqueued
      commands before backpressure is applied. Once this number is hit,
      enqueues will block until a message is dequeued.
      08db5173
  9. 23 Sep, 2016 1 commit
    • Ryan Senior's avatar
      (PDB-3026) Error log all level 2 retries · e9f165c4
      Ryan Senior authored
      With the new "quick" or L1 retries, we should be logging all L2 command
      processing failures at the error level. This change was made on master
      but lost in the merge into the cmd-proc-refactor branch. This commit
      puts that back in place.
      e9f165c4
  10. 22 Sep, 2016 3 commits
    • Ryan Senior's avatar
      (PDB-3026) Add test for node with ".json" in the name · 5b0a96cc
      Ryan Senior authored
      A basic test to ensure someone with a certname that includes ".json"
      isn't broken when the message goes into stockpile. This commit also
      includes some minor whitespace/indentation/comment changes
      5b0a96cc
    • Ryan Senior's avatar
      (PDB-3026) Changed import certname extract to a regex · 471684a1
      Ryan Senior authored
      Previously the code would use a somewhat complicated set of substring
      operations to extract the certname from the import file path. This
      commit switches that logic to a regex, which can also be used to detect
      the kind of command at a path. This was detected previously with a
      separate regex.
      471684a1
    • Dan Lidral-Porter's avatar
      (maint) Add `regex-quote` & `match-any-of` utils. · 8b84c454
      Dan Lidral-Porter authored
      Add `regex-quote` to the utils NS, which takes a string and returns
      a new string that quotes all of the special regex characters in the
      first string so it will be interpreted only literally.
      
      Add `match-any-of` which takes a collection of strings and characters,
      and returns a regex string with one capturing group that matches any of
      the members of the collection literally (using `regex-quote`).
      8b84c454
  11. 20 Sep, 2016 2 commits
  12. 15 Sep, 2016 1 commit
    • Karel Březina's avatar
      (PDB-3047) Improve event-counts and aggregate-event-counts endpoints on PE to... · de4bcd52
      Karel Březina authored
      (PDB-3047) Improve event-counts and aggregate-event-counts endpoints on PE to provide information about corrective changes.
      
      successes and noops counts were subdivided into intentional and corrective parts. So, successes and noops counts are replaced with intentional_successes, corrective_successes, intentional_noops, and corrective_noops. Events are mapped to the corresponding counts based on value of corrective_change flag.
      
      The response formats are unchanged for FOSS users.
      de4bcd52
  13. 14 Sep, 2016 3 commits
    • Ryan Senior's avatar
      (PDB-3026) Rename shovel to amq-migration · fd9cc98f
      Ryan Senior authored
      This commit just renames the ActiveMQ upgrade code from
      puppetlabs.puppetdb.cli.shovel to puppetlabs.puppetdb.amq-upgrader.
      fd9cc98f
    • Ryan Senior's avatar
      (PDB-3026) Remove the "catalog-facts-bash" config · b97b8a8b
      Ryan Senior authored
      This config was never intended to be released but was added to allow A/B
      testing of the feature along with the existing code. This feature tested
      well and found that it should be always enabled, so it no longer needs a
      config item.
      b97b8a8b
    • Ryan Senior's avatar
      (MAINT) Move remaining tests to new cmd testutils · 8e0d221c
      Ryan Senior authored
      This commit moves the remaining tests over to the command body coercion
      functions that convert from clojure/strings/etc to an InputStream like
      Stockpile expects. This commit also makes some other minor test and
      error messages changes.
      8e0d221c
  14. 09 Sep, 2016 4 commits
  15. 08 Sep, 2016 1 commit
    • Ryan Senior's avatar
      (PDB-3031) Separate stockpile open and load · 2eae4201
      Ryan Senior authored
      This commit opens the stockpile queue, then loads the command channel
      with existing entries in a background thread. This allows the startup
      process to continue while the commands are being re-enqueued in memory.
      
      This commit also includes a bump to stockpile 0.0.2 which includes fixes
      that allow opening a queue to occur separately from processing the
      existing messages.
      2eae4201
  16. 07 Sep, 2016 1 commit
  17. 02 Sep, 2016 1 commit
    • Ryan Senior's avatar
      (PDB-2734) Add stockpile upgrade code · 7628a208
      Ryan Senior authored
      This commit adds the ability to upgrade from an existing ActiveMQ queue
      to Stockpile. This happens automatically on startup. If an ActiveMQ
      queue is present, it moves all of the messages out of the queue and into
      Stockpile. It leaves a "lock" file in place so that there's no need to
      initialize the ActiveMQ broker and connections upon restart.
      7628a208
  18. 01 Sep, 2016 1 commit
    • Rob Browning's avatar
      (PDB-2993) Keep exceptions in cmdref :attempts · 5ae2c4d6
      Rob Browning authored
      Instead of the summary strings currently stored for each failed attempt,
      store {:exception ex :time stamp} so that consumers like the dlo can
      examine the exceptions directly.
      
      Move :attempts from :annotations to a top-level (optional) cmdref field.
      5ae2c4d6
  19. 31 Aug, 2016 3 commits
    • Rob Browning's avatar
      (PDB-2905) delay-message: remove vestigial arg · 16a8ccaf
      Rob Browning authored
      The exception argument is vestigial; remove it and update the relevant
      tests.
      16a8ccaf
    • Rob Browning's avatar
      (PDB-2905) annotate cmdref for all retries · 08e7a6e7
      Rob Browning authored
      08e7a6e7
    • Rob Browning's avatar
      (PDB-2905) Migrate dlo to stockpile · 8fcdf4db
      Rob Browning authored
      Migrate the dlo to stockpile and simplify it.  Stop periodically
      archiving messages, and don't pay attention to filesystem changes after
      startup.  Perform an initial scan for existing discards at startup, and
      only reflect pdb's changes in the metrics afterward.
      
      Store the discards via stockpile's discard command to a file matching
      the command's name in the pdb queue, and write information describing
      the failure to a corresponding *-err.txt file.  Until we rework the
      attempts to be exceptions, just write the strings we have.
      8fcdf4db