Skip to content

Commit 500367a

Browse files
dylburgerdylburger
authored
* Closes PipedreamHQ#3071 * Correcting example
1 parent 602e156 commit 500367a

3 files changed

Lines changed: 212 additions & 1 deletion

File tree

‎docs/docs/.vuepress/configs/sidebarConfig.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ const docsNav = [
3636
"/code/nodejs/working-with-files/",
3737
"/code/nodejs/using-data-stores/",
3838
"/code/nodejs/delay/",
39+
"/code/nodejs/rerun/",
3940
"/environment-variables/",
4041
"/code/nodejs/async/",
4142
"/code/nodejs/sharing-code/"

‎docs/docs/code/nodejs/delay/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ short_description: How to delay a workflow's execution with Node.js.
33
thumbnail: https://res.cloudinary.com/pipedreamin/image/upload/v1646841376/docs/icons/icons8-time-96_kupxpi.png
44
---
55

6-
# Delaying a workflow in Node.js
6+
# Delaying a workflow
77

88
<VideoPlayer title="Delaying Workflow Steps" url="https://www.youtube.com/embed/IBORwBnIZ-k" startAt="148" />
99

‎docs/docs/code/nodejs/rerun/README.md‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
---
2+
short_description: How to rerun a step execution in Node.js
3+
thumbnail: https://res.cloudinary.com/pipedreamin/image/upload/v1646841376/docs/icons/icons8-time-96_kupxpi.png
4+
---
5+
6+
# Pause, resume, and rerun a workflow
7+
8+
You can use `$.flow.suspend` and `$.flow.rerun` to pause a workflow and resume it later.
9+
10+
This is useful when you want to:
11+
12+
- Pause a workflow until someone manually approves it
13+
- Poll an external API until some job completes, and proceed with the workflow when it's done
14+
- Trigger an external API to start a job, pause the workflow, and resume it when the external API sends an HTTP callback
15+
16+
We'll cover all of these examples below.
17+
18+
[[toc]]
19+
20+
## `$.flow.suspend`
21+
22+
Use `$.flow.suspend` when you want to pause a workflow and proceed with the remaining steps only when manually approved or cancelled.
23+
24+
For example, you can suspend a workflow and send yourself a link to manually resume or cancel the rest of the workflow:
25+
26+
```javascript
27+
export default defineComponent({
28+
async run({ $ }) {
29+
const { resume_url, cancel_url } = $.flow.suspend()
30+
$.send.email({
31+
subject: "Please approve this important workflow",
32+
text: `Click here to approve the workflow: ${resume_url}, and cancel here: ${cancel_url}`,
33+
})
34+
// Pipedream suspends your workflow at the end of the step
35+
},
36+
})
37+
```
38+
39+
You'll receive an email like this:
40+
41+
<div>
42+
<img src="https://res.cloudinary.com/pipedreamin/image/upload/v1655272047/docs/approve-workflow_oc06k3.png" alt="approve this workflow" width="100%"/>
43+
</div>
44+
45+
And can resume or cancel the rest of the workflow by clicking on the appropriate link.
46+
47+
### `resume_url` and `cancel_url`
48+
49+
In general, calling `$.flow.suspend` returns a `cancel_url` and `resume_url` that lets you cancel or resume paused executions. Since Pipedream pauses your workflow at the _end_ of the step, you can pass these URLs to any external service before the workflow pauses. If that service accepts a callback URL, it can trigger the `resume_url` when its work is complete.
50+
51+
These URLs are specific to a single execution of your workflow. While the workflow is paused, you can load these in your browser or send any HTTP request to them:
52+
53+
- Sending an HTTP request to the `cancel_url` will cancel that execution
54+
- Sending an HTTP request to the `resume_url` will resume that execution
55+
56+
If you resume a workflow, any data sent in the HTTP request is passed to the workflow and returned in the `$resume_data` [step export](/workflows/steps/#step-exports) of the suspended step. For example, if you call `$.flow.suspend` within a step named `code`, the `$resume_data` export should contain the data sent in the `resume_url` request:
57+
58+
<div>
59+
<img src="https://res.cloudinary.com/pipedreamin/image/upload/v1655271815/docs/resume_data_lafhxr.png" alt="resume data step export" width="350px"/>
60+
</div>
61+
62+
### Default timeout of 24 hours
63+
64+
By default, `$.flow.suspend` will automatically resume the workflow after 24 hours. You can set your own timeout (in milliseconds) as the first argument:
65+
66+
```javascript
67+
export default defineComponent({
68+
async run({ $ }) {
69+
// 7 days
70+
const TIMEOUT = 1000 * 60 * 60 * 24 * 7
71+
$.flow.suspend(TIMEOUT)
72+
},
73+
})
74+
```
75+
76+
## `$.flow.rerun`
77+
78+
Use `$.flow.rerun` when you want to run a specific step of a workflow multiple times. This is useful when you need to start a job in an external API and poll for its completion, or have the service call back to the step and let you process the HTTP request within the step.
79+
80+
### Polling for the status of an external job
81+
82+
Sometimes you need to poll for the status of an external job until it completes. `$.flow.rerun` lets you rerun a specific step multiple times:
83+
84+
```javascript
85+
import axios from 'axios'
86+
87+
export default defineComponent({
88+
async run({ $ }) {
89+
const MAX_RETRIES = 3
90+
// 10 seconds
91+
const DELAY = 1000 * 10
92+
const { run } = $.context
93+
// $.context.run.runs starts at 1 and increments when the step is rerun
94+
if (run.runs === 1) {
95+
// $.flow.rerun(delay, context (discussed below), max retries)
96+
$.flow.rerun(DELAY, null, MAX_RETRIES)
97+
}
98+
else if (run.runs === MAX_RETRIES + 1) {
99+
throw new Error("Max retries exceeded")
100+
}
101+
else {
102+
// Poll external API for status
103+
const { data } = await axios({
104+
method: "GET",
105+
url: "https://example.com/status"
106+
})
107+
// If we're done, continue with the rest of the workflow
108+
if (data.status === "DONE") return data
109+
110+
// Else retry later
111+
$.flow.rerun(DELAY, null, MAX_RETRIES)
112+
}
113+
},
114+
})
115+
```
116+
117+
`$.flow.rerun` accepts the following arguments:
118+
119+
```javascript
120+
$.flow.rerun(
121+
delay, // The number of milliseconds until the step will be rerun
122+
context, // JSON-serializable data you need to pass between runs
123+
maxRetries, // The total number of times the step will rerun. Defaults to 10
124+
)
125+
```
126+
127+
### Accept an HTTP callback from an external service
128+
129+
When you trigger a job in an external service, and that service can send back data in an HTTP callback to Pipedream, you can process that data within the same step using `$.flow.retry`:
130+
131+
```javascript
132+
import axios from 'axios'
133+
134+
export default defineComponent({
135+
async run({ steps, $ }) {
136+
const TIMEOUT = 86400 * 1000
137+
const { run } = $.context
138+
// $.context.run.runs starts at 1 and increments when the step is rerun
139+
if (run.runs === 1) {
140+
const { cancel_url, resume_url } = $.flow.rerun(TIMEOUT, null, 1)
141+
142+
// Send resume_url to external service
143+
await axios({
144+
method: "POST",
145+
url: "your callback URL",
146+
data: {
147+
resume_url,
148+
cancel_url,
149+
}
150+
})
151+
}
152+
else if (run.runs === 2) {
153+
throw new Error("External service never completed job")
154+
}
155+
// When the external service calls back into the resume_url, you have access to
156+
// the callback data within $.context.run.callback_request
157+
else {
158+
const { callback_request } = run
159+
return callback_request
160+
}
161+
},
162+
})
163+
```
164+
165+
### Passing `context` to `$.flow.rerun`
166+
167+
Within a Node.js code step, `$.context.run.context` contains the `context` passed from the prior call to `rerun`. This lets you pass data from one run to another. For example, if you call:
168+
169+
```javascript
170+
$.flow.rerun(1000, { hello: "world" })
171+
```
172+
173+
`$.context.run.context` will contain:
174+
175+
<div>
176+
<img src="https://res.cloudinary.com/pipedreamin/image/upload/v1655274732/docs/Screen_Shot_2022-06-14_at_11.32.06_PM_dmzgkh.png" alt="resume data step export" width="250px"/>
177+
</div>
178+
179+
### `maxRetries`
180+
181+
By default, `maxRetries` is **10**.
182+
183+
When you exceed `maxRetries`, the workflow proceeds to the next step. If you need to handle this case with an exception, `throw` an error from the step:
184+
185+
```javascript
186+
export default defineComponent({
187+
async run({ $ }) {
188+
const MAX_RETRIES = 3
189+
const { run } = $.context
190+
if (run.runs === 1) {
191+
$.flow.rerun(1000, null, MAX_RETRIES)
192+
}
193+
else if (run.runs === MAX_RETRIES + 1) {
194+
throw new Error("Max retries exceeded")
195+
}
196+
},
197+
})
198+
```
199+
200+
## Behavior when testing
201+
202+
When you're building a workflow and test a step with `$.flow.suspend` or `$.flow.rerun`, it will not suspend the workflow, and you'll see a message like the following:
203+
204+
> Workflow execution canceled — this may be due to `$.flow.suspend()` usage (not supported in test)
205+
206+
These functions will only suspend and resume when run in production.
207+
208+
## Invocations when using `suspend` / `rerun`
209+
210+
Each time workflows are resumed, Pipedream charges [an invocation](/pricing/#invocations). For example, when you call `$.flow.suspend`, you're charged an invocation for the initial event data, and another invocation when you resume the request.

0 commit comments

Comments
 (0)