n8nworkflows.xyz/workflows/Automatically track certification changes with ScrapeGraphAI, GitLab and Rocket.Chat-12154/readme-12154.md

Automatically track certification changes with ScrapeGraphAI, GitLab and Rocket.Chat

https://n8nworkflows.xyz/workflows/automatically-track-certification-changes-with-scrapegraphai--gitlab-and-rocket-chat-12154

Automatically track certification changes with ScrapeGraphAI, GitLab and Rocket.Chat

disclaimer Le texte fourni provient exclusivement d’un workflow automatisé réalisé avec n8n, un outil d’intégration et d’automatisation. Ce traitement respecte strictement les politiques de contenu en vigueur et ne contient aucun élément illégal, offensant ou protégé. Toutes les données manipulées sont légales et publiques.

1. Workflow Overview

Title: Automatically track certification changes with ScrapeGraphAI, GitLab and Rocket.Chat
Workflow name (internal): Certification Requirement Tracker with Rocket.Chat and GitLab

Purpose:
Runs on a schedule to scrape certification requirement pages, compare the latest scraped data to the previously stored version in GitLab, and then either (a) update GitLab and notify a chat channel when changes are detected, or (b) log a “no change” event in GitLab.

Primary use cases:

• Compliance monitoring for certification renewals and updated requirements
• Audit-friendly tracking of changes over time using GitLab history/issues
• Automated alerting when official certification bodies change requirements

1.1 Logical Blocks (by node dependency)

1. Input & Iteration
   • Schedule trigger → build list of certifications → process each in sequence
2. Scraping
   • Scrape each target webpage into structured JSON
3. Error Gate
   • Detect scrape errors and (intended) alert path vs. continue path
4. State Retrieval & Comparison
   • Fetch previous JSON from GitLab → merge current+previous → compare in Code
5. Conditional Actions
   • If changed: prepare file → commit updated JSON to GitLab → craft message → send chat message
   • If not changed: create a GitLab issue as a log entry

Important mismatch: despite the title/sticky notes mentioning Rocket.Chat, the final messaging node is a Slack node (“Send a message”). Also, the IF branch for scrape errors is not wired to any alert node.

---

2. Block-by-Block Analysis

Block 1 — Input & Iteration

Overview:
Starts the workflow on a fixed schedule, generates the list of certification URLs to track, then iterates over them one-by-one.

Nodes involved:

• Daily Trigger
• Certification URL Config
• Split In Batches

Node: Daily Trigger

• Type / role: Schedule Trigger (n8n-nodes-base.scheduleTrigger) — workflow entry point.
• Configuration: Runs every 24 hours (hoursInterval: 24).
• Inputs/outputs: No inputs; outputs to Certification URL Config.
• Version notes: typeVersion 1.1.
• Failure modes / edge cases:
  • If n8n instance timezone differs from expectations, “daily” timing may drift from business hours.
  • If the instance is down at trigger time, scheduled execution may be missed depending on n8n setup.

Node: Certification URL Config

• Type / role: Code (n8n-nodes-base.code) — defines tracked certifications (IDs + URLs).
• Configuration choices: Hardcoded list in JavaScript returning two items:
  • certId: "pmp", url: https://www.pmi.org/certifications/pmp
  • certId: "cissp", url: https://www.isc2.org/Certifications/CISSP
• Key variables:
  • Produces item.json.certId and item.json.url used downstream (scraping and GitLab path).
• Inputs/outputs: Input from trigger; output to Split In Batches.
• Version notes: typeVersion 2.
• Failure modes / edge cases:
  • Invalid URL / blocked page / bot protection can later cause scraping failure.
  • Duplicate certId would cause overwriting the same GitLab file.

Node: Split In Batches

• Type / role: Split In Batches — processes one certification per iteration.
• Configuration choices: Default options (batch size defaults to 1 in typical n8n behavior).
• Inputs/outputs: Input from URL config; output to Scrape Requirement Data.
• Version notes: typeVersion 3.
• Failure modes / edge cases:
  • If you expect looping to continue automatically, ensure the node is configured/connected properly for “next batch” behavior. In this workflow JSON, only the forward path is shown; no explicit “continue” connection is present.

---

Block 2 — Scraping

Overview:
Scrapes each certification webpage via ScrapeGraphAI using a natural-language prompt and returns structured JSON fields.

Nodes involved:

• Scrape Requirement Data

Node: Scrape Requirement Data

• Type / role: ScrapeGraphAI (n8n-nodes-scrapegraphai.scrapegraphAi) — extracts structured fields from a web page.
• Configuration choices:
  • websiteUrl: ={{ $json.url }}
  • userPrompt: “Extract the certification name, full requirement description, last updated date, and renewal interval in years… Return JSON with keys: certName, requirementText, lastUpdated, renewalIntervalYears.”
• Inputs/outputs: Input from Split In Batches; output to Scrape Error?.
• Version notes: typeVersion 1 (community node; ensure installed and compatible with your n8n version).
• Failure modes / edge cases:
  • Website blocks scraping, requires JS rendering, or rate-limits.
  • LLM extraction inconsistencies: missing keys, wrong types (e.g., renewalIntervalYears as string).
  • Node may return an error field or throw an execution error (the workflow only checks $json.error, which may not exist if the node fails hard).

---

Block 3 — Error Gate

Overview:
Checks whether the scraping step produced an error flag. If no error is detected, proceeds to fetch previous data from GitLab and compare.

Nodes involved:

• Scrape Error?

Node: Scrape Error?

• Type / role: IF — conditional routing based on a boolean expression.
• Configuration choices:
  • Condition: ={{ $json.error }} equals true
• Connections:
  • True branch (error): not connected to anything (no alert is actually sent).
  • False branch (no error): connected to Fetch Previous Data and also directly to Merge Current & Previous (input index 0).
• Version notes: typeVersion 2.
• Failure modes / edge cases:
  • If ScrapeGraphAI returns errors differently (e.g., nested error object), $json.error may be undefined and evaluate false, incorrectly continuing.
  • Because the true branch is not wired, scrape errors will silently stop for that item with no notification.
  • The IF node’s false branch sends data to both Fetch and Merge; this is intentional to provide “current” to the merge’s input 0.

---

Block 4 — State Retrieval & Comparison

Overview:
Loads the previous JSON snapshot for the same certification from GitLab, merges it with the current scrape result, and computes whether anything changed.

Nodes involved:

• Fetch Previous Data
• Merge Current & Previous
• Detect Changes

Node: Fetch Previous Data

• Type / role: GitLab — fetch a file from a GitLab repository.
• Configuration choices:
  • Resource: file
  • Operation: get
  • filePath: ={{ '/certifications/' + $json.certId + '.json' }}
  • (Branch not specified in parameters for “get”; behavior depends on node defaults—often defaults to the default branch.)
• Inputs/outputs: Input from IF (no-error path); output to Merge Current & Previous (input index 1).
• Version notes: typeVersion 1.
• Failure modes / edge cases:
  • Auth/permission error to repo.
  • File doesn’t exist on first run → GitLab node may throw an error rather than returning empty content (depends on implementation). The subsequent Code node expects previous.json to be content; missing file should be handled (currently only handles JSON parse errors, not GitLab 404 failures).
  • Wrong repository/project selected in credentials.

Node: Merge Current & Previous

• Type / role: Merge — combine two inputs into one item by position.
• Configuration choices:
  • Mode: mergeByPosition
  • Input 0: current scraped item (from IF false branch)
  • Input 1: previous file content (from GitLab get)
• Inputs/outputs: Outputs merged item to Detect Changes.
• Version notes: typeVersion 2.
• Failure modes / edge cases:
  • If one side produces zero items (e.g., GitLab get failed), merge may output nothing or misalign items.
  • If multiple items pass through unexpectedly, position-based merge can pair the wrong items.

Node: Detect Changes

• Type / role: Code — parses previous JSON and compares to current scrape.
• Configuration choices (interpreted):
  • Reads both inputs via $input.all() and assumes [current, previous].
  • scraped = current.json
  • prevContent = previous.json
  • Attempts JSON.parse(prevContent); if parsing fails, treats as null.
  • Change detection: changed = !prevParsed || JSON.stringify(scraped) !== JSON.stringify(prevParsed)
  • Returns merged output:
    • All scraped fields
    • certId: current.json.certId
    • changed boolean
    • prevData (parsed previous JSON or null)
• Connections: Output to Requirement Changed?
• Version notes: typeVersion 2.
• Failure modes / edge cases:
  • GitLab “get file” content format: Many GitLab file APIs return file metadata and base64 content, not raw JSON string. If previous.json is an object (not a string), JSON.parse(prevContent) will fail.
  • JSON.stringify comparison is order-sensitive for object keys; ScrapeGraphAI could reorder fields or add noise, producing “changed” even when semantically identical.
  • scraped likely contains url and certId from upstream? Actually ScrapeGraphAI might output only extracted keys and drop certId; the code re-adds certId from current.json.certId, but if ScrapeGraphAI overwrote/removed it, current.json.certId could be missing (depends on node output behavior).

---

Block 5 — Conditional Actions (Storage + Notifications)

Overview:
Branches based on changed. If true, updates the JSON file in GitLab and sends a chat message. If false, logs an issue in GitLab.

Nodes involved:

• Requirement Changed?
• Prepare GitLab File
• Save Updated Requirement
• Craft Alert Message
• Send a message
• Log No-Change Issue

Node: Requirement Changed?

• Type / role: IF — checks $json.changed.
• Configuration: Boolean condition ={{ $json.changed }} equals true.
• Connections:
  • True → Prepare GitLab File
  • False → Log No-Change Issue
• Version notes: typeVersion 2.
• Failure modes / edge cases:
  • If changed is missing (undefined), it will evaluate false and route to “no change”.

Node: Prepare GitLab File

• Type / role: Set — intended to construct filePath, file content, and commit message.
• Configuration choices: Not configured (no fields defined).
• Connections: To Save Updated Requirement.
• Version notes: typeVersion 3.
• Failure modes / edge cases:
  • As-is, Save Updated Requirement expects filePath and commitMsg in $json but they are never set here; GitLab edit will fail.
  • Also missing is the actual file content parameter (GitLab “edit file” requires content); this node does not provide it.

Node: Save Updated Requirement

• Type / role: GitLab — edits (creates/updates) a file in repo.
• Configuration choices:
  • Resource: file
  • Operation: edit
  • Branch: main
  • filePath: ={{ $json.filePath }}
  • commitMessage: ={{ $json.commitMsg }}
  • (No visible “content” parameter set in this JSON; typically required.)
• Connections: To Craft Alert Message
• Version notes: typeVersion 1.
• Failure modes / edge cases:
  • Missing/empty filePath or commitMsg due to Set node not configured.
  • Missing file content parameter will cause API validation failure.
  • Permission/branch protection may reject commits to main.

Node: Craft Alert Message

• Type / role: Set — intended to prepare a message payload for chat notification.
• Configuration choices: Not configured (no fields defined).
• Connections: To Send a message
• Version notes: typeVersion 3.
• Failure modes / edge cases:
  • With no fields set, the Slack node may send an empty/default message (or fail, depending on required fields).

Node: Send a message

• Type / role: Slack (n8n-nodes-base.slack) — posts a message to Slack.
• Configuration choices: Only otherOptions present; channel/text not shown (likely relies on node defaults or missing config).
• Connections: Terminal node.
• Version notes: typeVersion 2.3.
• Failure modes / edge cases:
  • Missing required Slack parameters (channel, text) will fail.
  • Credential/auth errors; workspace/channel permissions.

Note: This contradicts the workflow title and sticky notes that say Rocket.Chat. If Rocket.Chat is required, replace this node with the Rocket.Chat node.

Node: Log No-Change Issue

• Type / role: GitLab — creates an issue used as an audit log for “no change”.
• Configuration choices:
  • Title: No change for {{$json.certId}} on {{$now.toFormat('yyyy-LL-dd')}}
  • Labels: empty
  • Assignee IDs: empty
  • (Resource/operation not explicitly shown in the snippet, but implied “Issue → Create” by the parameters.)
• Connections: Terminal node.
• Version notes: typeVersion 1.
• Failure modes / edge cases:
  • Repeated daily issues may spam the project; consider using commits or a single rolling issue instead.
  • If certId missing, title becomes malformed.

---

3. Summary Table

Node Name	Node Type	Functional Role	Input Node(s)	Output Node(s)	Sticky Note
Workflow Overview	Sticky Note	Documentation block			(content) How it works + setup steps (ScrapeGraphAI/GitLab/Rocket.Chat)
Section – Data Collection	Sticky Note	Documentation block			(content) Data Collection & Scraping description
Section – Change Detection	Sticky Note	Documentation block			(content) Comparison Logic description
Section – Actions	Sticky Note	Documentation block			(content) Conditional Actions & Storage description
Daily Trigger	Schedule Trigger	Periodic workflow start	—	Certification URL Config	Section – Data Collection: Data Collection & Scraping (applies to this cluster)
Certification URL Config	Code	Defines certId/url list	Daily Trigger	Split In Batches	Section – Data Collection: Data Collection & Scraping
Split In Batches	SplitInBatches	Serializes processing per cert	Certification URL Config	Scrape Requirement Data	Section – Data Collection: Data Collection & Scraping
Scrape Requirement Data	ScrapeGraphAI	Scrape & extract structured data	Split In Batches	Scrape Error?	Section – Data Collection: Data Collection & Scraping
Scrape Error?	IF	Gate on scrape error flag	Scrape Requirement Data	Fetch Previous Data; Merge Current & Previous	Section – Data Collection: Data Collection & Scraping
Fetch Previous Data	GitLab (File Get)	Load previous snapshot from repo	Scrape Error? (false)	Merge Current & Previous	Section – Change Detection: Comparison Logic
Merge Current & Previous	Merge	Pair current scrape with previous snapshot	Scrape Error? (false); Fetch Previous Data	Detect Changes	Section – Change Detection: Comparison Logic
Detect Changes	Code	Parse/compare; set changed flag	Merge Current & Previous	Requirement Changed?	Section – Change Detection: Comparison Logic
Requirement Changed?	IF	Branch on changed	Detect Changes	Prepare GitLab File; Log No-Change Issue	Section – Actions: Conditional Actions & Storage
Prepare GitLab File	Set	Build filePath/content/commit message (intended)	Requirement Changed? (true)	Save Updated Requirement	Section – Actions: Conditional Actions & Storage
Save Updated Requirement	GitLab (File Edit)	Commit updated JSON to GitLab	Prepare GitLab File	Craft Alert Message	Section – Actions: Conditional Actions & Storage
Craft Alert Message	Set	Build chat alert payload (intended)	Save Updated Requirement	Send a message	Section – Actions: Conditional Actions & Storage
Send a message	Slack	Send notification to chat	Craft Alert Message	—	Section – Actions: Conditional Actions & Storage (note: says Rocket.Chat but node is Slack)
Log No-Change Issue	GitLab (Issue Create)	Audit log when no change	Requirement Changed? (false)	—	Section – Actions: Conditional Actions & Storage

---

4. Reproducing the Workflow from Scratch

1. Create a new workflow

   • Name it: “Certification Requirement Tracker with Rocket.Chat and GitLab” (or your preferred name).

2. Add Schedule Trigger

   • Node: Schedule Trigger
   • Set interval: every 24 hours (or adjust cadence).

3. Add Code node: “Certification URL Config”

   • Node: Code
   • Paste and adapt logic to output items like:
     • certId (unique identifier)
     • url (certification page URL)
   • Ensure each returned item is { json: { certId, url } }.

4. Connect: Schedule Trigger → Certification URL Config

5. Add Split In Batches

   • Node: Split In Batches
   • Keep defaults (or set Batch Size = 1 explicitly).

6. Connect: Certification URL Config → Split In Batches

7. Add ScrapeGraphAI node: “Scrape Requirement Data”

   • Node: ScrapeGraphAI
   • Credentials: add ScrapeGraphAI credential in n8n Credentials.
   • Set Website URL to expression: {{$json.url}}
   • Set Prompt to extract:
     • certName
     • requirementText
     • lastUpdated
     • renewalIntervalYears

8. Connect: Split In Batches → Scrape Requirement Data

9. Add IF node: “Scrape Error?”

   • Condition (boolean): {{$json.error}} equals true

10. Connect: Scrape Requirement Data → Scrape Error?

11. Add GitLab node: “Fetch Previous Data”

• Node: GitLab
• Credentials: create a GitLab credential (PAT/OAuth) with access to your target project/repo.
• Resource: File
• Operation: Get
• File path expression: {{ '/certifications/' + $json.certId + '.json' }}
• Ensure project/repo is set in the node/credential as required.

12. Connect: Scrape Error? (false branch) → Fetch Previous Data

13. Add Merge node: “Merge Current & Previous”

• Mode: Merge By Position

14. Connect current input: Scrape Error? (false branch) → Merge Current & Previous (Input 1 / index 0 in JSON)

15. Connect previous input: Fetch Previous Data → Merge Current & Previous (Input 2 / index 1)

16. Add Code node: “Detect Changes”

• Implement:
  • Parse previous file content into JSON
  • Compare with current scrape
  • Output: changed, certId, and current fields
• If your GitLab “get file” returns base64 content or a wrapper object, decode/extract content first (adjust code accordingly).

17. Connect: Merge Current & Previous → Detect Changes

18. Add IF node: “Requirement Changed?”

• Condition (boolean): {{$json.changed}} equals true

19. Connect: Detect Changes → Requirement Changed?

20. Changed = true branch: prepare commit

• Add Set node: “Prepare GitLab File”
• Configure fields you must set (minimum):
  • filePath: {{ '/certifications/' + $json.certId + '.json' }}
  • commitMsg: e.g. {{ 'Update requirements for ' + $json.certId + ' on ' + $now.toFormat('yyyy-LL-dd') }}
  • content: {{ JSON.stringify($json, null, 2) }} (or a subset containing only scraped fields)
• Connect: Requirement Changed? (true) → Prepare GitLab File

21. Add GitLab node: “Save Updated Requirement”

• Resource: File
• Operation: Edit (or “Create/Update” depending on node capabilities)
• Branch: main
• filePath: {{$json.filePath}}
• Commit message: {{$json.commitMsg}}
• Content/body: map from {{$json.content}} (exact parameter name depends on the GitLab node UI)
• Connect: Prepare GitLab File → Save Updated Requirement

22. Add Set node: “Craft Alert Message”

• Create a text field (and optionally channel) describing:
  • cert name/id
  • what changed (you may extend Detect Changes to generate a diff)
• Connect: Save Updated Requirement → Craft Alert Message

23. Add chat node

• If you truly want Rocket.Chat: add Rocket.Chat node and map message text/channel.
• If using Slack (as in provided JSON): add Slack node “Send a message”, configure:
  • Channel
  • Text: {{$json.text}}
• Connect: Craft Alert Message → Send a message

24. Changed = false branch: log audit event

• Add GitLab node: “Log No-Change Issue”
• Resource: Issue
• Operation: Create
• Title: No change for {{$json.certId}} on {{$now.toFormat('yyyy-LL-dd')}}
• Connect: Requirement Changed? (false) → Log No-Change Issue

25. (Recommended) Handle scrape errors

• Add a Rocket.Chat/Slack node on Scrape Error? (true) branch to alert with certId, url, and error details.
• Without this, scrape errors produce no notifications.

26. Enable workflow

• Turn the workflow Active and verify GitLab/Chat messages on the next scheduled run.

---

5. General Notes & Resources

Note Content	Context or Link
Sticky notes describe Rocket.Chat alerts, but the implemented messaging node is Slack (“Send a message”).	Align the node choice (Rocket.Chat vs Slack) with intended behavior.
Scrape error branch is not connected to any alert action.	Add a notification node on the IF true branch to avoid silent failures.
Change detection uses JSON.stringify equality.	Consider semantic comparison (ignore whitespace, normalize dates, compare selected fields only).
GitLab file “get” response format may not be raw JSON text.	You may need to extract/decode content before JSON.parse.