Skip to content

Commit 4cb78c9

Browse files
PijukatelPijukatel
authored
docs(openapi): Extract and re-use parameters defined in paths (#2324)
## Summary - Add Agents directives about changes in the OpenAPI spec - Re-use previously extracted parameters in remaining paths - Extract and re-use non-unique remaining parameters where possible - Remove incorrect pagination query parameters in [apify-api/openapi/paths/webhooks/webhooks.yaml](https://docs.apify.com/api/v2/webhooks-post) - Log-related options `stream` and `download` should be optional - Extract and re-use `AbortedRunExample` ## Motivation - Preparation for adding undocumented redispatch endpoints, which will re-use the error schemas and parameters. - Make the specification more maintainable ## Issues Partially implements: #2286
1 parent 089e0cf commit 4cb78c9

File tree

78 files changed

+1131
-3552
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+1131
-3552
lines changed

‎AGENTS.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,17 @@ Add code samples by creating files in `apify-api/openapi/code_samples/{javascrip
5151
- Decorator auto-detects and adds `x-codeSamples` property
5252
- Missing samples are logged during build
5353

54+
### OpenAPI specification changes
55+
56+
- Prefer re-use of existing objects via `$ref` over duplication. Reusable components can be found in `/openapi/components`.
57+
- Components most suitable for re-use are:
58+
- Request parameters and path parameters defined in `/openapi/components/parameters`
59+
- Request/response schemas defined in `/openapi/components/schemas`
60+
- Explicit non-automatic examples defined in `/openapi/components/examples`
61+
- When changing files in `/openapi/paths` look for opportunities to extract shared duplicate objects into re-usable components saved in `/openapi/components`.
62+
- When adding new endpoints, check first if any existing path is similar and if yes, try to re-use same components. If by adding new paths you create new duplication, try to extract it into a new components and reference it instead.
63+
- Prefer automatically generated examples from schema over explicit examples.
64+
5465
### Theme system
5566

5667
Uses `@apify/docs-theme` package - a shared theme across all 6+ documentation repos. Don't modify theme files directly. Changes to the theme propagate via CI to all projects.

‎apify-api/openapi/components/examples/AbortedRunExample.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
data:
2+
id: HG7ML7M8z78YcAPEB
3+
actId: janedoe~my-actor
4+
userId: BPWZBd7Z9c746JAng
5+
actorTaskId: rANaydYhUxjsnA3oz
6+
startedAt: "2019-11-30T07:34:24.202Z"
7+
finishedAt: "2019-12-12T09:30:12.202Z"
8+
status: ABORTED
9+
statusMessage: Actor was aborted
10+
isStatusMessageTerminal: true
11+
meta:
12+
origin: WEB
13+
clientIp: 172.234.12.34
14+
userAgent: Mozilla/5.0 (iPad)
15+
stats:
16+
inputBodyLen: 240
17+
migrationCount: 0
18+
restartCount: 0
19+
resurrectCount: 1
20+
memAvgBytes: 35914228.4
21+
memMaxBytes: 38244352
22+
memCurrentBytes: 0
23+
cpuAvgUsage: 0.00955965
24+
cpuMaxUsage: 3.1546
25+
cpuCurrentUsage: 0
26+
netRxBytes: 2652
27+
netTxBytes: 1338
28+
durationMillis: 26239
29+
runTimeSecs: 26.239
30+
metamorph: 0
31+
computeUnits: 0.0072886
32+
options:
33+
build: latest
34+
timeoutSecs: 300
35+
memoryMbytes: 1024
36+
diskMbytes: 2048
37+
buildId: 7sT5jcggjjA9fNcxF
38+
exitCode: 0
39+
generalAccess: RESTRICTED
40+
defaultKeyValueStoreId: eJNzqsbPiopwJcgGQ
41+
defaultDatasetId: wmKPijuyDnPZAPRMk
42+
defaultRequestQueueId: FL35cSF7jrxr3BY39
43+
storageIds:
44+
datasets:
45+
default: wmKPijuyDnPZAPRMk
46+
keyValueStores:
47+
default: eJNzqsbPiopwJcgGQ
48+
requestQueues:
49+
default: FL35cSF7jrxr3BY39
50+
isContainerServerReady: false
51+
gitBranchName: master
52+
usage:
53+
ACTOR_COMPUTE_UNITS: 3
54+
DATASET_READS: 4
55+
DATASET_WRITES: 4
56+
KEY_VALUE_STORE_READS: 5
57+
KEY_VALUE_STORE_WRITES: 3
58+
KEY_VALUE_STORE_LISTS: 5
59+
REQUEST_QUEUE_READS: 2
60+
REQUEST_QUEUE_WRITES: 1
61+
DATA_TRANSFER_INTERNAL_GBYTES: 1
62+
DATA_TRANSFER_EXTERNAL_GBYTES: 3
63+
PROXY_RESIDENTIAL_TRANSFER_GBYTES: 34
64+
PROXY_SERPS: 3
65+
usageTotalUsd: 0.2654
66+
usageUsd:
67+
ACTOR_COMPUTE_UNITS: 0.072
68+
DATASET_READS: 0.0004
69+
DATASET_WRITES: 0.0002
70+
KEY_VALUE_STORE_READS: 0.0006
71+
KEY_VALUE_STORE_WRITES: 0.002
72+
KEY_VALUE_STORE_LISTS: 0.004
73+
REQUEST_QUEUE_READS: 0.005
74+
REQUEST_QUEUE_WRITES: 0.02
75+
DATA_TRANSFER_INTERNAL_GBYTES: 0.0004
76+
DATA_TRANSFER_EXTERNAL_GBYTES: 0.0002
77+
PROXY_RESIDENTIAL_TRANSFER_GBYTES: 0.16
78+
PROXY_SERPS: 0.0006

‎apify-api/openapi/components/parameters/actor-run-options/maxItems.yaml‎

Lines changed: 0 additions & 15 deletions
This file was deleted.

‎apify-api/openapi/components/parameters/actor-run-options/maxTotalChargeUsd.yaml‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎apify-api/openapi/components/parameters/datasetItemsParameters.yaml‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
format:
2+
name: format
3+
in: query
4+
description: |
5+
Format of the results, possible values are: `json`, `jsonl`, `csv`, `html`, `xlsx`, `xml` and `rss`. The default value is `json`.
6+
style: form
7+
explode: true
8+
schema:
9+
type: string
10+
example: json
11+
12+
clean:
13+
name: clean
14+
in: query
15+
description: |
16+
If `true` or `1` then the API endpoint returns only non-empty items and skips hidden fields (i.e. fields starting with the # character).
17+
The `clean` parameter is just a shortcut for `skipHidden=true` and `skipEmpty=true` parameters.
18+
Note that since some objects might be skipped from the output, that the result might contain less items than the `limit` value.
19+
style: form
20+
explode: true
21+
schema:
22+
type: boolean
23+
example: false
24+
25+
limit:
26+
name: limit
27+
in: query
28+
description: Maximum number of items to return. By default there is no limit.
29+
style: form
30+
explode: true
31+
schema:
32+
type: number
33+
format: double
34+
35+
fields:
36+
name: fields
37+
in: query
38+
description: |
39+
A comma-separated list of fields which should be picked from the items, only these fields will remain in the resulting record objects.
40+
Note that the fields in the outputted items are sorted the same way as they are specified in the `fields` query parameter.
41+
You can use this feature to effectively fix the output format.
42+
style: form
43+
explode: true
44+
schema:
45+
type: string
46+
example: "myValue,myOtherValue"
47+
48+
omit:
49+
name: omit
50+
in: query
51+
description: A comma-separated list of fields which should be omitted from the items.
52+
style: form
53+
explode: true
54+
schema:
55+
type: string
56+
example: "myValue,myOtherValue"
57+
58+
unwind:
59+
name: unwind
60+
in: query
61+
description: |
62+
A comma-separated list of fields which should be unwound, in order which they should be processed. Each field should be either an array or an object.
63+
If the field is an array then every element of the array will become a separate record and merged with parent object.
64+
If the unwound field is an object then it is merged with the parent object.
65+
If the unwound field is missing or its value is neither an array nor an object and therefore cannot be merged with a parent object then the item gets preserved as it is.
66+
Note that the unwound items ignore the `desc` parameter.
67+
style: form
68+
explode: true
69+
schema:
70+
type: string
71+
example: "myValue,myOtherValue"
72+
73+
flatten:
74+
name: flatten
75+
in: query
76+
description: |
77+
A comma-separated list of fields which should transform nested objects into flat structures.
78+
79+
For example, with `flatten="foo"` the object `{"foo":{"bar": "hello"}}` is turned into `{"foo.bar": "hello"}`.
80+
81+
The original object with properties is replaced with the flattened object.
82+
style: form
83+
explode: true
84+
schema:
85+
type: string
86+
example: myValue
87+
88+
attachment:
89+
name: attachment
90+
in: query
91+
description: |
92+
If `true` or `1` then the response will define the `Content-Disposition:
93+
attachment` header, forcing a web browser to download the file rather
94+
than to display it. By default this header is not present.
95+
style: form
96+
explode: true
97+
schema:
98+
type: boolean
99+
example: true
100+
101+
delimiter:
102+
name: delimiter
103+
in: query
104+
description: |
105+
A delimiter character for CSV files, only used if `format=csv`. You
106+
might need to URL-encode the character (e.g. use `%09` for tab or `%3B`
107+
for semicolon). The default delimiter is a simple comma (`,`).
108+
style: form
109+
explode: true
110+
schema:
111+
type: string
112+
example: ;
113+
114+
bom:
115+
name: bom
116+
in: query
117+
description: |
118+
All text responses are encoded in UTF-8 encoding. By default, the
119+
`format=csv` files are prefixed with the UTF-8 Byte Order Mark (BOM), while `json`, `jsonl`, `xml`, `html` and `rss` files are not.
120+
121+
If you want to override this default behavior, specify `bom=1` query parameter to include the BOM or `bom=0` to skip it.
122+
style: form
123+
explode: true
124+
schema:
125+
type: boolean
126+
example: false
127+
128+
xmlRoot:
129+
name: xmlRoot
130+
in: query
131+
description: |
132+
Overrides default root element name of `xml` output. By default the root element is `items`.
133+
style: form
134+
explode: true
135+
schema:
136+
type: string
137+
example: items
138+
139+
xmlRow:
140+
name: xmlRow
141+
in: query
142+
description: |
143+
Overrides default element name that wraps each page or page function result object in `xml` output. By default the element name is `item`.
144+
style: form
145+
explode: true
146+
schema:
147+
type: string
148+
example: item
149+
150+
skipHeaderRow:
151+
name: skipHeaderRow
152+
in: query
153+
description: If `true` or `1` then header row in the `csv` format is skipped.
154+
style: form
155+
explode: true
156+
schema:
157+
type: boolean
158+
example: true
159+
160+
skipHidden:
161+
name: skipHidden
162+
in: query
163+
description: |
164+
If `true` or `1` then hidden fields are skipped from the output, i.e. fields starting with the `#` character.
165+
style: form
166+
explode: true
167+
schema:
168+
type: boolean
169+
example: false
170+
171+
skipEmpty:
172+
name: skipEmpty
173+
in: query
174+
description: |
175+
If `true` or `1` then empty items are skipped from the output.
176+
177+
Note that if used, the results might contain less items than the limit value.
178+
style: form
179+
explode: true
180+
schema:
181+
type: boolean
182+
example: false
183+
184+
simplified:
185+
name: simplified
186+
in: query
187+
description: |
188+
If `true` or `1` then, the endpoint applies the `fields=url,pageFunctionResult,errorInfo`
189+
and `unwind=pageFunctionResult` query parameters. This feature is used to emulate simplified results provided by the
190+
legacy Apify Crawler product and it's not recommended to use it in new integrations.
191+
style: form
192+
explode: true
193+
schema:
194+
type: boolean
195+
example: false
196+
197+
descDataset:
198+
name: desc
199+
in: query
200+
description: |
201+
By default, results are returned in the same order as they were stored.
202+
To reverse the order, set this parameter to `true` or `1`.
203+
style: form
204+
explode: true
205+
schema:
206+
type: boolean
207+
example: true
208+
209+
skipFailedPages:
210+
name: skipFailedPages
211+
in: query
212+
description: |
213+
If `true` or `1` then, the all the items with errorInfo property will be skipped from the output.
214+
215+
This feature is here to emulate functionality of API version 1 used for the legacy Apify Crawler product and it's not recommended to use it in new integrations.
216+
style: form
217+
explode: true
218+
schema:
219+
type: boolean
220+
example: false

‎apify-api/openapi/components/parameters/logOptions.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
stream:
2+
name: stream
3+
in: query
4+
description: |
5+
If `true` or `1` then the logs will be streamed as long as the run or build is running.
6+
required: false
7+
style: form
8+
explode: true
9+
schema:
10+
type: boolean
11+
example: false
12+
13+
download:
14+
name: download
15+
in: query
16+
description: |
17+
If `true` or `1` then the web browser will download the log file rather than open it in a tab.
18+
required: false
19+
style: form
20+
explode: true
21+
schema:
22+
type: boolean
23+
example: false

0 commit comments

Comments
 (0)